diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 63b347c..186e673 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1496,3 +1496,201 @@ thing that ever closes a requirement. chart default rather than set explicitly — see the follow-up in `platform/longhorn/`. - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI. + +--- + +## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.** + +Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put +`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes** +and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down). +Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold +prod bootstrap than it was here. + +### What happened + +The HelmRelease config digest tells the whole story: + +``` +v8 05:49:39 digest e83bf695… <- before +v9 05:50:11 digest eebd6c88… <- the bad render +v10 05:55:23 digest e83bf695… <- identical to v8 again +``` + +The config changed and then changed **back to byte-identical**. Nothing in git differed +semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus +`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets +`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady +state is identical. The damage happened in a transient intermediate state. + +### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations + +- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization. +- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`** + Kustomization, via `postBuild.substituteFrom: cluster-vars`. +- There is **no dependency ordering between defining a variable and using it.** + +So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`, +which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an +undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm +upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` — +which only the Cilium agent itself can translate. The agent was the thing restarting, so its own +`config` init container could not reach the API server: + +``` +level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443 + error="dial tcp 10.43.0.1:443: i/o timeout" +level=fatal msg="Build config failed" +``` + +Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the +correct value, digest returned to `e83bf695…`, and all three agents came up. + +`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires +the ConfigMap to *exist*, not that every referenced key is defined. + +### Why this matters much more for prod + +On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the +race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**, +there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may +never come up at all — with no healthy peer to recover from. This is a plausible +bring-up-blocker for `clusters/prod`. + +### The fix — make substitution FAIL CLOSED + +`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict +substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail +the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine +("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts +this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile +failure*. + +**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization +that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in +embedded scripts. The safe sequence, before flipping it: +1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in + `cluster-vars` or correctly escaped as `$${VAR}`. +2. Then enable the gate, and let CI + a dev reconcile prove it. + +Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and +reference it in a later one**, so the definition is always reconciled before its first use. + +### Operational note worth keeping + +A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the +`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is +set, and the agent that translates that ClusterIP is the one being replaced. Keep +`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time. + +--- + +## 2026-09-01 — RESEARCH: Flux substitution + HelmRelease best practice. **Measured semantics + ranked plan.** + +Three parallel research passes (substitution/ordering, HelmRelease/CNI, and a full repo audit), +prompted by the incident above. Two of my own earlier conclusions were WRONG and are corrected +here. Everything below marked *measured* was run against flux 2.4.0 / the live cluster; everything +else is cited inference. + +### THE SUBSTITUTION SEMANTICS TABLE (measured — trust this over prose elsewhere) + +| form | behaviour | +|---|---| +| `${DEFINED}` | substituted | +| `${UNDEFINED}` | **silently empty** (non-strict) · fails loudly (strict) | +| `${UNDEFINED:-d}` | **default applied at reconcile time — `--strict` does NOT catch it** | +| `${VAR:?msg}` | renders **the message as the value** — a non-empty WRONG value. ANTI-PATTERN. | +| `${VAR:=192.168.1.50}` | silently ships the dev value to prod. ANTI-PATTERN. | +| `$${VAR}` / `$${VAR:-d}` / `$${ARR[0]}` | literal — **escaping is TOTAL** | +| `$VAR` / `$(cmd)` | untouched, no escaping needed | +| `${...}` / `${a b}` / `${ARR[0]}` unescaped | **hard parse error — fails the whole Kustomization** | + +Flux exempts default-providing operators (`-`, `:-`, `+`, `:+`, `=`, `:=`, `?`, `:?`) from strict +mode by design ("they intentionally handle the unset case"). **That is why strict mode would never +have caught the gitea/nextcloud dead-guard bug**, and why a targeted check was required. + +**Doctrine corrected:** `clusters/base/apps-vms.yaml` claimed an indexed expansion is fatal +"escaped or not". Measured: escaping is total; only the UNESCAPED form is fatal. Fixed in place. + +**Comment nuance:** `kustomize build` strips *structural* YAML comments before envsubst sees them, +so those are usually harmless. But `#` lines INSIDE a block scalar are data and DO reach envsubst, +and a `$`-carrying body comment was observed surviving into built `apps/wave-1`. Do not rely on +stripping — escape in comments too. (An earlier claim in this log that a YAML comment "nearly broke +every merge" was overstated for the structural case; the hazard is real for block-scalar bodies.) + +### CORRECTION 1 (mine) — `StrictPostBuildSubstitutions` alone is INSUFFICIENT + +It fixes nothing about ordering; it converts "render a corrupt value and ship it" into "fail loudly +and keep last-good". **The actual ordering fix is one line**, verified against +`kustomization_controller.go :: checkDependencies` at our exact version (v1.4.0): `dependsOn` is +**revision-gated**, not merely Ready-gated — when the dependency shares a GitRepository, the +dependent cannot reconcile revision N until the dependency has APPLIED revision N. + +`platform` is the only Kustomization in the graph with no `dependsOn`; every other consumer chains +transitively through it. So: + +```yaml +# clusters/base/platform.yaml +spec: + dependsOn: + - name: flux-system # cluster-vars must be applied at THIS revision first +``` + +**Cost, recorded honestly:** the root becomes a gate — a malformed manifest anywhere under +`clusters/dev` now stops the whole graph where today it would not. Fail-closed, consistent with +doctrine, but a real blast-radius change. No deadlock risk: the root runs `wait: false`. + +Rejected alternative: moving `cluster-vars` into the consuming Kustomization is **strictly worse** — +`SubstituteVariables` reads the ConfigMap from the API at build time, *before* the Kustomization +applies its own resources, so it would always read the PREVIOUS value. That converts an occasional +race into a permanent one-reconcile lag. + +### CORRECTION 2 (an agent's) — the safety machinery DID fire + +A research pass concluded that Helm considered the bad release healthy (because +`updateStrategy.rollingUpdate.maxUnavailable: 2` on 3 nodes means `expectedReady = 1`), and +therefore remediation never engaged. **Measured on the live cluster: that is wrong.** The Helm +release secrets show **v9 = `failed`**; Helm's wait DID fail and the release was remediated, which +is why it self-healed. `maxUnavailable: 2 of desired 3` IS confirmed live — so the real harm was +blast radius (2 of 3 agents down at once), not a missed detection. + +### Verified, and worth knowing + +- **Flux 2.4.0 is OUT OF SUPPORT** — the project supports the last three minors; 2.9.x is current. +- **Strict substitution became DEFAULT-ON** in kustomize-controller v1.9.0. So upgrading Flux flips + this behaviour whether or not we opt in — the variable audit is a prerequisite for the upgrade + regardless of today's decision. +- **Cilium's `values.schema.json` permits `k8sServiceHost: ""`** — no `minLength`, no `pattern`. + Helm was never going to reject the bad render. +- **Cilium docs make `k8sServiceHost` REQUIRED under kube-proxy replacement**, precisely because the + environment exports the API ClusterIP and there is no kube-proxy to provision it. +- **The Cilium chart ships zero test templates**, so `spec.test` is useless for it. A post-upgrade + check must live outside Helm. +- `valuesFrom` fails HARD on a missing key (verified in `chartutil/values.go`) unlike postBuild's + silent empty — but only on a MISSING key, not an empty value, and it cannot cross namespaces + (`cluster-vars` is in `flux-system`; the Cilium HelmRelease is in `kube-system`). +- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** It compares the cluster against the + Helm storage manifest — during the incident that manifest WAS the bad render, so drift correction + would have actively re-asserted the broken value. + +### RANKED PLAN + +| P | Action | State | +|---|---|---| +| **P0** | `platform.dependsOn: [flux-system]` — closes the race completely | **PENDING — operator call** (blast-radius change) | +| **P0** | Escape the dead OIDC guards; renderer now refuses the class | ✅ DONE | +| **P1** | Substitution-safety L0 gate, unconditional in `ci/l0.sh` | ✅ DONE — 102 assertions | +| **P1** | Enable `StrictPostBuildSubstitutions=true` | **READY — proven safe**, all 11 paths exit 0 under `flux envsubst --strict`. Sequence AFTER P0. | +| **P2** | `updateStrategy.rollingUpdate.maxUnavailable: 1` on Cilium, as a new **[parity]** field in BOTH the HelmRelease and the Ansible j2 | PENDING — bounds blast radius to one node | +| **P3** | Split Cilium into its own Kustomization (`wait: true`, `interval: 1h`, `timeout: 10m`); `platform` dependsOn it | PENDING — gives `flux suspend kustomization cilium` as an emergency valve | +| **P4** | Upgrade Flux 2.4.0 → 2.9.x | PENDING — out of support; brings strict-by-default | +| **P5** | Cilium MINOR bumps become manual (Renovate PR → human runs the documented pre-flight → merge). Never automerge a CNI minor. | PENDING — process | +| — | Record `${VAR:?}` / `${VAR:=}` as anti-patterns | ✅ DONE (table above) | +| — | *Not recommended*: drift detection on Cilium; `spec.test` on Cilium; `k8sServiceHost: auto` | — | + +### Follow-ups surfaced, not yet actioned +- 6 dead `cluster-vars` knobs; `BASE_DOMAIN` and `CLUSTER_ENV` look like genuine rot. +- `platform/{cilium,longhorn,velero,oauth2-proxy}/values.yaml` reference mirrors carry decorative + `${...}` that can silently drift from the real HelmRelease. +- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time + out while a legitimate Longhorn upgrade is still running. diff --git a/apps/vms/unifi/30-virtualmachine.yaml b/apps/vms/unifi/30-virtualmachine.yaml index dc9a963..ae0d159 100644 --- a/apps/vms/unifi/30-virtualmachine.yaml +++ b/apps/vms/unifi/30-virtualmachine.yaml @@ -92,10 +92,13 @@ spec: # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData. # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits - # as a literal ${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone - # by envsubst and need no escaping. NEVER write an unescaped ${...} here, and never write - # ${ARR[0]} even escaped-adjacent: envsubst hard-fails on `[` ("missing closing brace") - # and would break the whole apps-vms apply. + # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone + # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write + # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace") + # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against + # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal + # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not" + # wording in clusters/base/apps-vms.yaml overstated it. userData: |- #cloud-config hostname: unifi diff --git a/apps/vms/unifi/50-service-l4.yaml b/apps/vms/unifi/50-service-l4.yaml index 4ba925a..a380ee6 100644 --- a/apps/vms/unifi/50-service-l4.yaml +++ b/apps/vms/unifi/50-service-l4.yaml @@ -27,7 +27,7 @@ # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53). # Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for -# exactly this token; every shell ${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...} +# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...} # accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool. # # REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an diff --git a/apps/wave-0/gitea.yaml b/apps/wave-0/gitea.yaml index 067949e..359b9df 100644 --- a/apps/wave-0/gitea.yaml +++ b/apps/wave-0/gitea.yaml @@ -127,8 +127,8 @@ spec: command: - /bin/sh - -c - - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"$GITEA_ADMIN_USERNAME@home.local\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name\ - \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n" + - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"$GITEA_ADMIN_USERNAME@home.local\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\ + \ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n" volumes: - name: data persistentVolumeClaim: diff --git a/apps/wave-1/nextcloud.yaml b/apps/wave-1/nextcloud.yaml index 6f68a40..b1a0f3d 100644 --- a/apps/wave-1/nextcloud.yaml +++ b/apps/wave-1/nextcloud.yaml @@ -125,8 +125,9 @@ spec: command: - /bin/sh - -c - - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra\ - \ positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# ` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n" + - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"\ + [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# ` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"\ + $ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n" volumes: - name: data persistentVolumeClaim: diff --git a/bootstrap/flux/README.md b/bootstrap/flux/README.md index 90465ce..1824ad0 100644 --- a/bootstrap/flux/README.md +++ b/bootstrap/flux/README.md @@ -11,11 +11,69 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux ## Files | File | Purpose | |---|---| -| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. | +| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. | +| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. | | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). | | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). | | `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). | +## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster + +Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git +and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01): + +``` +$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}' +{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system", + "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"} +# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1 +``` + +No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under +`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux +documents for hand-installed clusters ([Customize Flux +manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)): + +``` +flux install --export … -> /gotk-components.yaml # regenerated from the pinned flux CLI +components/kustomization.yaml -> /kustomization.yaml # the only hand-authored part (in git) +kubectl kustomize | kubectl apply -f - +``` + +Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION` +(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations, +more feature gates) is another entry in `components/kustomization.yaml`. + +### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic) + +> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the +> controller Deployment is outside Flux's inventory. Someone must run step 1. + +Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1: + +```bash +# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root +d=$(mktemp -d) +flux install --export --namespace=flux-system \ + --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml" +cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml" +kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict +kubectl -n flux-system rollout status deploy/kustomize-controller +``` + +**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any +reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing +is lost. Confirm afterwards: + +```bash +kubectl -n flux-system get deploy kustomize-controller \ + -o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true +flux get kustomizations -A # all 11 must return to Ready +``` + +**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or +simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off). + ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1) The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker): diff --git a/bootstrap/flux/bootstrap.sh b/bootstrap/flux/bootstrap.sh index 893b19c..33e35c7 100755 --- a/bootstrap/flux/bootstrap.sh +++ b/bootstrap/flux/bootstrap.sh @@ -69,16 +69,59 @@ for bin in flux kubectl; do command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; } done -# --- 1. install Flux controllers --------------------------------------------- +# --- 1. install Flux controllers (rendered through ./components) -------------- # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates, # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly. -if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then - echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})" - flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}" +# +# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently: +# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined +# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see +# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md). +# +# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the +# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux +# documents for hand-installed clusters ("Customize Flux manifests"): +# +# flux install --export > gotk-components.yaml # exact manifests `flux install` would apply +# kustomize overlay = ./components/kustomization.yaml (checked in — the ONLY hand-authored part) +# kubectl apply -k +# +# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION, +# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is +# a temp dir — the repo tree is never written to. +# +# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO +# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so. +echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}" +flux_install_args=(--namespace="${FLUX_NAMESPACE}") +[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}") + +if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then + echo " WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED." + echo " StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN." + flux install "${flux_install_args[@]}" else - echo "==> [1/3] flux install (default controllers only)" - flux install --namespace="${FLUX_NAMESPACE}" + components_dir="${script_dir}/components" + [ -f "${components_dir}/kustomization.yaml" ] || { + echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2 + echo " (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2 + exit 1 + } + render_dir="$(mktemp -d)" + trap 'rm -rf "${render_dir}"' EXIT + echo " rendering flux install --export through ${components_dir}" + flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml" + cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml" + # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux + # version bump) — better a hard stop than silently installing without the feature gate. + kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml" + grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || { + echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2 + echo " Check ./components/kustomization.yaml against this flux CLI version." >&2 + exit 1 + } + kubectl apply -f "${render_dir}/rendered.yaml" fi # --- 2. verify the out-of-band age key Secret exists ------------------------- diff --git a/bootstrap/flux/components/kustomization.yaml b/bootstrap/flux/components/kustomization.yaml new file mode 100644 index 0000000..0a428ec --- /dev/null +++ b/bootstrap/flux/components/kustomization.yaml @@ -0,0 +1,64 @@ +--- +# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments). +# +# WHY THIS DIRECTORY EXISTS +# ========================= +# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the +# reconciliation graph is authored by hand in clusters/base, and the controllers are installed +# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01: +# +# kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}' +# -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1 +# +# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no +# existing mechanism to customize them, and no amount of editing under clusters/** can reach them. +# +# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" — +# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests +# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result. +# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported +# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the +# repo, so the manifests can never drift from the pinned FLUX_VERSION +# (infra/devcontainer/versions.env) the way a vendored copy would. +# +# Every future controller customization (vertical scaling, tolerations, extra feature gates, +# --concurrent) belongs HERE, as another entry under `patches:`. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + # Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in. + - gotk-components.yaml +patches: + # ------------------------------------------------------------------------------------------- + # StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED. + # + # Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the + # EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium + # HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this + # gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state. + # + # PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with + # `kubectl kustomize --load-restrictor LoadRestrictionsNone | flux envsubst --strict` and + # exited 0, with a negative control reproducing the incident. The permanent guard that keeps it + # true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/. + # + # NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) — + # Flux exempts them by design. That class is what the L0 gate covers; the two are complementary. + # + # This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in + # kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in. + # + # JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic + # list, so a strategic-merge patch would REPLACE every arg `flux install` generates + # (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election) + # and silently drop any the CLI adds in a future version. + - target: + group: apps + version: v1 + kind: Deployment + name: kustomize-controller + namespace: flux-system + patch: |- + - op: add + path: /spec/template/spec/containers/0/args/- + value: --feature-gates=StrictPostBuildSubstitutions=true diff --git a/catalog/README.md b/catalog/README.md index 4b5912a..271de76 100644 --- a/catalog/README.md +++ b/catalog/README.md @@ -28,6 +28,16 @@ inherit every guarantee automatically. `auth.groups`; `native`/`none` are documented exceptions only. - Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`. +- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a + `lifecycle` postStart script, an `env` value — lands under `apps/wave-/`, and every + `apps-wave-` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs + envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the + *shell* is replaced with the empty string before the container ever starts (this silently killed + gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and + need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and + written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml). + `render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode + cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms). ## Validate before commit — single command, no cluster (R-TEST-02) ```bash diff --git a/catalog/gitea.yaml b/catalog/gitea.yaml index ab07fba..f7a2885 100644 --- a/catalog/gitea.yaml +++ b/catalog/gitea.yaml @@ -109,7 +109,19 @@ lifecycle: echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2 { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true } - if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then + # NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild. + # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE + # time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not + # a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE + # — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on + # every boot since 8738871 (2026-08-27). SSO kept working only because the login source was + # already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all. + # Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design. + # `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell. + # Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the + # `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke. + # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py. + if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)" else # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200). diff --git a/catalog/nextcloud.yaml b/catalog/nextcloud.yaml index f28aba1..3723d89 100644 --- a/catalog/nextcloud.yaml +++ b/catalog/nextcloud.yaml @@ -111,7 +111,19 @@ lifecycle: # every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27. OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \ || echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2 - if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then + # NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild. + # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE + # time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is + # not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS + # TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never + # installing/enabling user_oidc nor registering the provider. SSO kept working only because + # the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with + # NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator + # forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion + # for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why + # `--clientsecret-env` and the bare uses below always worked and only the guard broke. + # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py. + if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)" exit 0 fi diff --git a/catalog/render.py b/catalog/render.py index 1873554..992f531 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -29,6 +29,7 @@ conformance harness stays stdlib-only and only shells out to this script + diffs from __future__ import annotations import argparse +import re import sys from pathlib import Path @@ -936,6 +937,124 @@ def r_virtualmachine(s: dict) -> dict: } +# ============================================================================= +# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that +# Flux's envsubst would silently eat before the workload ever runs. +# ============================================================================= +# Every manifest this renderer writes lands under apps/wave-/, and EVERY apps-wave- +# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see +# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered +# document — including any shell script carried verbatim in a `lifecycle` hook or an env +# value — at reconcile time, long before a container sees it. The escape contract is stated +# in clusters/base/apps-vms.yaml and proven with `flux envsubst`: +# +# `${VAR}` -> SUBSTITUTED (empty when unset) => a shell brace must be written `$${VAR}`, +# which envsubst emits back as `${VAR}`. +# `$VAR` -> left alone => bare shell vars need NO escaping. +# `$(cmd)` -> left alone => command substitution needs NO escaping. +# `${A[0]}` -> HARD ERROR ("missing closing brace"), failing the whole Kustomization. +# +# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC +# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var, +# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the +# registration entirely. Flux's STRICT substitution mode cannot catch this class: it +# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is +# exactly the form a defensive shell script uses. So the renderer checks it itself. +CLUSTER_VARS_PATH = ( + Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml" +) + +# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N +# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is +# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms +# (`${A[0]}`) are caught too rather than skipped. +_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}") +_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z") + +_CLUSTER_VARS_CACHE: dict = {} + + +def cluster_vars_keys(path: Path | None = None) -> set: + """The declared cluster-vars keys — the ONLY names envsubst may legitimately expand. + + Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the + renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e. + FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell. + """ + key = str(path or CLUSTER_VARS_PATH) + if key not in _CLUSTER_VARS_CACHE: + try: + doc = yaml.safe_load(Path(key).read_text()) + data = doc.get("data") if isinstance(doc, dict) else None + _CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set() + except Exception: # pragma: no cover - absent/malformed => fail closed + _CLUSTER_VARS_CACHE[key] = set() + return _CLUSTER_VARS_CACHE[key] + + +def substitution_violations(text: str, known: set | None = None) -> list: + """Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key. + + Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only + when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug: + a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that + envsubst will blank, or an indexed form that hard-fails the whole Kustomization. + """ + keys = cluster_vars_keys() if known is None else known + out = [] + for lineno, line in enumerate(text.splitlines(), start=1): + for m in _BRACED_EXPANSION_RE.finditer(line): + if len(m.group(1)) % 2 == 0: + continue # even run of `$` => escaped; envsubst emits it back literally + expr = m.group(2) + if _PLAIN_IDENT_RE.match(expr): + if expr in keys: + continue # a declared cluster-vars knob, intentionally substituted + reason = ( + f"'{expr}' is not a declared cluster-vars key — envsubst will replace " + "it with the EMPTY STRING" + ) + elif "[" in expr: + reason = ( + "indexed expansion — envsubst hard-fails with 'missing closing brace' " + "and the whole Kustomization stops reconciling" + ) + else: + reason = ( + "shell expansion (default-operator/parameter form) — envsubst eats it " + "at reconcile time and STRICT mode exempts this form, so nothing else " + "will catch it" + ) + out.append((lineno, m.group(0), reason)) + return out + + +def assert_substitution_safe(text: str, origin: str = "") -> None: + """Refuse to emit a manifest carrying an unescaped braced expansion (see above). + + Raises ValueError listing every offending token. The fix is almost always to escape the + shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the + value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml + and write it as a bare `${KNOB}`. + """ + bad = substitution_violations(text) + if not bad: + return + lines = [ + f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's " + "postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:" + ] + for lineno, token, reason in bad: + lines.append(f" line {lineno}: {token} -> {reason}") + lines.append( + " FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal " + "`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob " + "must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. " + "Contract: clusters/base/apps-vms.yaml." + ) + raise ValueError("\n".join(lines)) + + # ============================================================================= # Top-level render + emit # ============================================================================= @@ -1026,8 +1145,14 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None) return docs -def emit(docs: list) -> str: - """Deterministic multi-document YAML text (golden-diffable).""" +def emit(docs: list, origin: str = "") -> str: + """Deterministic multi-document YAML text (golden-diffable). + + Every emitted document is checked against the substitution contract before it is handed + back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs + envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises + ValueError rather than emitting such a manifest. + """ parts = [] for comment, resource in docs: body = yaml.dump( @@ -1037,7 +1162,9 @@ def emit(docs: list) -> str: width=4096, ) parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n") - return "".join(parts) + text = "".join(parts) + assert_substitution_safe(text, origin) + return text def load_sd(path: Path) -> dict: @@ -1076,7 +1203,9 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N src_dir = catalog_dir if catalog_dir is not None else path.parent rev = build_reverse_map(_load_catalog(src_dir)) inbound = rev.get(sd["name"], []) - return emit(render(sd, image_automation=image_automation, inbound=inbound)) + return emit( + render(sd, image_automation=image_automation, inbound=inbound), origin=str(path) + ) # ============================================================================= @@ -1729,6 +1858,90 @@ def _selftest() -> int: check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"], "wave kustomization must list exactly the wave's resources") + # 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose + # shell script carries an unescaped `${...}`. Every rendered manifest lands under a + # Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before + # the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]` + # -> `[ -z "" ]`, always true, whole registration block skipped on every boot). + known = {"BASE_DOMAIN", "GATEWAY_VIP"} + # allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution. + check( + substitution_violations("host: app.${BASE_DOMAIN}", known) == [], + "a bare declared cluster-vars key must be allowed (that is what substitution is FOR)", + ) + # allowed: escaped shell braces, and the forms envsubst leaves alone. + check( + substitution_violations('if [ -z "$${X:-}" ]; then', known) == [], + "an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)", + ) + check( + substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [], + "bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged", + ) + # `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live. + check( + len(substitution_violations("$$${X:-}", known)) == 1, + "an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)", + ) + # flagged: the exact regression — a default-operator form STRICT mode deliberately exempts. + guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known) + check( + len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}", + "an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)", + ) + # flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error). + check( + len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1, + "an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)", + ) + idx = substitution_violations("echo ${PIPESTATUS[0]}", known) + check( + len(idx) == 1 and "missing closing brace" in idx[0][2], + "an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)", + ) + # the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the + # bug RAISES rather than writing a manifest whose guard is dead on arrival. + bad_sd = { + "name": "subbad", + "source": {"image": "a:1"}, + "lifecycle": { + "postStart": { + "exec": { + "command": [ + "/bin/sh", + "-c", + 'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi', + ] + } + } + }, + } + try: + emit(render(bad_sd), origin="") + check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion") + except ValueError as exc: + check( + "SUBBAD_SECRET" in str(exc) and "$${" in str(exc), + "the refusal must name the offending token AND state the `$${...}` fix", + ) + # the ESCAPED form of the same hook renders normally (the fix must be emittable). + good_sd = dict(bad_sd, name="subgood") + good_sd["lifecycle"] = { + "postStart": { + "exec": { + "command": [ + "/bin/sh", + "-c", + 'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi', + ] + } + } + } + check( + "$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin=""), + "the escaped `$${...}` form must render (and reach the shell as `${...}`)", + ) + if failures: for f in failures: sys.stderr.write(f"SELFTEST FAIL: {f}\n") diff --git a/ci/l0.sh b/ci/l0.sh index d4dc4cc..efcedda 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -100,6 +100,43 @@ else fi fi +# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ---------- +# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it +# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still +# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes +# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every +# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED +# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail +# a legitimate manifest, and that no expansion silently empties today. +# +# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the +# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys +# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/ +# attributes no requirement, so the requirement-scoped selection would filter this check +# out of exactly the change that introduces the bug. The invariant is whole-repo (same +# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation. +# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a +# sub-second offline scan with no side effects, so the duplication is not worth avoiding. +SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}" +log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})" +if [ ! -f "$SUBST_CHECK" ]; then + skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green" + summary "- subst-safety: **SKIPPED** (check missing)" + if [ "$rc" -eq 0 ]; then rc=3; fi +else + subst_rc=0 + bash "$SUBST_CHECK" || subst_rc=$? + case "$subst_rc" in + 0) ok "Flux postBuild substitution safety passed" + summary "- subst-safety: **passed**" ;; + 3) skip "substitution check SKIPPED (clusters/ absent) — not a pass" + summary "- subst-safety: **SKIPPED** (nothing to audit)" + if [ "$rc" -eq 0 ]; then rc=3; fi ;; + *) err "Flux postBuild substitution safety FAILED" + summary "- subst-safety: **FAILED**"; rc=1 ;; + esac +fi + case "$rc" in 0) ok "L0 stage complete" ;; 3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;; diff --git a/clusters/base/platform.yaml b/clusters/base/platform.yaml index 1f3dec6..02251e1 100644 --- a/clusters/base/platform.yaml +++ b/clusters/base/platform.yaml @@ -13,6 +13,33 @@ spec: timeout: 5m prune: true # drift correction: delete resources removed from git (R-GITOPS-03) wait: true # block `apps` until platform is healthy + # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium). + # + # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root + # `flux-system` Kustomization (clusters//sync.yaml), not by this one. Without an edge here, + # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a + # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING. + # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease. + # + # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against + # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version): + # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile + # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what + # writes a newly-added `cluster-vars` knob — so this one edge closes the race. + # + # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting + # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty, + # velero) chains transitively through it. One edge therefore protects the whole graph. + # + # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest + # ANYWHERE under the root's path (clusters/, which includes clusters/base) fails the root + # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge + # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken + # entrypoint should stop the line, not let consumers render against a half-applied variable set. + # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never + # health-gates its own children (see clusters/dev/sync.yaml). + dependsOn: + - name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars sourceRef: kind: GitRepository name: flux-system diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 5b8b0f9..ec34c1a 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -5,11 +5,144 @@ # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit. +# +# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD *** +# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution- +# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml` +# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would +# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing +# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml +# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT +# done here: prod is design-only and its root object is the prod bring-up owner's call.) apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base + - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: + # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) --- + # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets. + # The `sops-age` Secret is created OUT OF BAND on the prod cluster + # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined. + # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave. + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: platform + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-0 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-1 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-2 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: oauth2-proxy + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: headlamp + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: crafty + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: velero + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) --- + # + # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. *** + # + # Two independent reasons, both consequences of the split-horizon design: + # 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's + # GitOps source lives inside the very cluster slated for decommission. Gitea must move to + # prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. + # 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml + # answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is + # 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve + # gitea.home.keithjackson.dev to itself and never reach a repository. + # + # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster + # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the + # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7. + # + # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev. + # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH + # CLUSTERS. There is no promotion step and no soak window — dev and prod move together. + - patch: |- + apiVersion: source.toolkit.fluxcd.io/v1 + kind: GitRepository + metadata: + name: flux-system + namespace: flux-system + spec: + url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git + ref: + branch: main - target: kind: ConfigMap name: cluster-vars diff --git a/clusters/prod/sync.yaml b/clusters/prod/sync.yaml new file mode 100644 index 0000000..9130eee --- /dev/null +++ b/clusters/prod/sync.yaml @@ -0,0 +1,32 @@ +--- +# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml). +# +# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars +# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object +# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs +# its own sync.yaml -> ./clusters/prod"; this is it. +# +# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained +# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent +# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no +# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does +# not exist, and NOTHING downstream would ever reconcile. +# +# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches +# Ready on apply success without health-gating its children (platform does its own wait/health +# gating). Do not "harden" this to true. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: flux-system + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + sourceRef: + kind: GitRepository + name: flux-system + path: ./clusters/prod diff --git a/conformance/run.sh b/conformance/run.sh index 2d9efcc..6321768 100755 --- a/conformance/run.sh +++ b/conformance/run.sh @@ -57,17 +57,15 @@ FORCE_NO_CLUSTER=0 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the # exemption is explicit and reviewable, not silent. -# R-ENV-02 (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in -# coverage.md). The coarse path->requirement map keys -# clusters/* AND ci/* to it and the R-ENV-02 doc token -# appears in platform/kustomization.yaml's header, so ANY -# infra/overlay/CI edit (e.g. enabling a component, bumping -# SPEC_PIN) would otherwise false-block on it. -# R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/* -# path-map keys to it, so editing any ci/ file (SPEC_PIN -# adoption included) would false-block. -# These are visible GAPs, not hidden — removing them from this list is the right move the -# moment WS-TEST adds their verifying tests. +# +# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted +# because they were unscaffolded while the coarse path->requirement map keys clusters/* +# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real +# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the +# substitution mechanism that IS the "authored once, promoted" contract, so they are +# covered rather than exempt and the completeness gate can hold them to it. That test +# names its own limits in its header: it proves the STATIC property (no manifest renders +# an undefined variable), not the live-cluster half of either requirement. # # FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header # tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in @@ -83,7 +81,7 @@ FORCE_NO_CLUSTER=0 # R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra") # but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no # longer a gap and has been removed from this list; the coverage report counts it as covered. -DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01" +DOCUMENTED_GAPS="" usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } @@ -108,7 +106,11 @@ log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; } # --- Coverage report is always produced (cluster-independent, R-TEST-04) -------------- gen_coverage() { - local flags=(--allow-gap "${DOCUMENTED_GAPS}") + # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no + # requirement is exempt, and handing the flag an empty string would be a silent no-op + # that reads like an exemption in the CLI log. + local flags=() + [ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}") [ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict) # A tagged subset run is also the merge gate: the touched requirements MUST be covered. [ -n "${REQS}" ] && flags+=(--require "${REQS}") diff --git a/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml b/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml new file mode 100644 index 0000000..b048116 --- /dev/null +++ b/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml @@ -0,0 +1,60 @@ +# ============================================================================= +# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism: +# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each +# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes +# the empty string while the Kustomization still reports Ready — which emptied Cilium's +# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01). +# +# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom, +# WHEN each `spec.path` is resolved to the exact file set it renders (following +# kustomization.yaml `resources:` transitively, not globbing the repo), +# THEN every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml +# or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default +# operator on an undefined name (the silent-empty class strict mode does NOT catch); +# no unescaped expansion is unparseable by envsubst (which fails the WHOLE +# Kustomization); every path rendering a knob actually substitutes cluster-vars; +# every overlay-patched knob has a base default; and the check is wired into the +# merge gate as a blocking, unconditionally-run stage. +# +# This is the prerequisite proof for enabling kustomize-controller's +# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw +# Test wraps the SAME script. +# +# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests +# a path includes — a deliberate superset of the post-`kustomize build` stream. It does +# not render, does not prove the feature gate is enabled on the live kustomize-controller, +# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice +# only: what git says a Kustomization renders is fully determined, with no variable +# silently emptied on the way in. The live-state half is L2, untested today. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: flux-postbuild-substitution + labels: + conformance.home.k3s/R-ENV-02: "true" + conformance.home.k3s/R-GITOPS-01: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every variable a substituted Flux path renders is defined in cluster-vars or + escaped, so post-build substitution cannot silently empty a value and enabling + StrictPostBuildSubstitutions cannot fail a legitimate manifest. +spec: + steps: + - name: audit-every-substituted-flux-path + description: >- + Resolve each postBuild.substituteFrom Kustomization to the files it renders and + assert every ${...} is defined or escaped, then assert the gate wires it blocking. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/flux-postbuild-substitution/check.sh b/conformance/tests/l0-static/flux-postbuild-substitution/check.sh new file mode 100755 index 0000000..c05c725 --- /dev/null +++ b/conformance/tests/l0-static/flux-postbuild-substitution/check.sh @@ -0,0 +1,369 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — flux-postbuild-substitution +# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)." +# — spec docs/02-environments-and-provisioning.md) +# R-GITOPS-01 ("All desired state in git, reconciled by Flux." +# — spec docs/05-gitops-and-delivery.md) +# +# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN +# -------------------------------------------------------- +# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml` +# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders, +# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an +# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports +# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put +# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that +# fails OPEN"). "Authored once, promoted" is therefore only true while every token a +# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in +# the repo asserted that. This check does. +# +# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain, +# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption): +# ${DEFINED} -> substituted with the cluster-vars value. +# ${UNDEFINED} -> EMPTY STRING, silently. `--strict` / the cluster-wide +# StrictPostBuildSubstitutions feature gate turns this into a +# loud Kustomization failure ("variable not set (strict mode)"). +# ${UNDEFINED:-d} -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does +# NOT catch this (it exempts default-operator forms by design), +# so a shell guard written `[ -z "${X:-}" ]` inside a rendered +# manifest is pre-resolved to `[ -z "" ]` and its else-branch +# becomes unreachable in the container. Silent, and permanent. +# $${VAR}, $${V:-d}, $${ARR[0]} -> emitted as the LITERAL ${...} for a downstream +# shell/cloud-init consumer. Escaping is total: it neutralises +# even the indexed form. +# $VAR, $(cmd) -> left alone by envsubst; they need no escaping. +# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse +# variable name" / "missing closing brace") which fails the +# WHOLE Kustomization — every resource in the path, not just +# the offending one. +# The escape contract this enforces is the one already documented in +# `clusters/base/apps-vms.yaml`; this check is that comment made executable. +# +# GIVEN the repository as authored, +# WHEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom +# is resolved to the exact file set its `spec.path` renders (by following +# kustomization.yaml `resources:` transitively — a path, not a glob), +# THEN (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`; +# (2) no UNESCAPED expansion carries a default/length/substitution operator on an +# undefined name (the silent-empty class strict mode does not catch); +# (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) — +# that fails the entire Kustomization, so it is fatal wherever it appears; +# (4) every Kustomization that renders a cluster-vars token actually declares +# `substituteFrom` a ConfigMap named cluster-vars (else the literal token +# ships into the cluster); +# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap — +# the "add the knob with a base default BEFORE referencing it" rule that the +# incident's interim discipline depends on; +# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and +# ci/gate.sh sets the merge block when the l0 stage fails. +# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build` +# preserves body comments into the stream envsubst reads (verified: a `$`-carrying +# comment survives into the built apps/wave-1 output), so a braced expression in a +# comment is a live hazard, not prose. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check): +# - It does NOT render the manifests. It scans the SOURCE files a path includes, +# which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize +# drops some comments). A finding in a comment may not reach envsubst TODAY; it is +# still reported, because whether a given comment survives the build is a kustomize +# implementation detail that changes with the pinned version, and the failure it +# causes is total (assertion 3). The end-to-end proof — `kustomize build | +# flux envsubst --strict` for every substituted path — needs kustomize + flux, +# which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh +# l0), so it cannot run here without permanently SKIPPING. It was run by hand and +# recorded in DECISIONS-LOG; this check is the reproducible static half. +# - It does NOT prove the cluster's live state matches git, nor that the +# StrictPostBuildSubstitutions feature gate is actually enabled on +# kustomize-controller. Both are cluster facts (L2), untested here. Tagging +# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders +# is fully determined, with no variable silently emptied on the way in. +# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool, +# that an overlay's LAN block is self-consistent). Presence, not correctness. +# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure: +# several are deliberately reference-only and say so in cluster-vars.yaml +# (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so +# failing on them would be a false block on documented intent. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +# Honest SKIP (never green) if the thing under test is not present at all. +if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then + echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2 + exit 3 +fi + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit of Flux postBuild substitution safety. Pure standard library. + +No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are +documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization +`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one +definition of "what a Flux path actually renders" in the harness. +""" +from __future__ import annotations + +import os +import re +import sys +from pathlib import Path + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree +sys.path.insert(0, str(ROOT / "conformance" / "lib")) +import status as st # noqa: E402 — reuse the walk, don't fork it + +FAILURES: list[str] = [] +ADVISORIES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +# ------------------------------------------------------------------ cluster-vars +# Parse contract: `data:` block of a single ConfigMap, one ` KEY: "value"` per line +# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and +# continuation-comment lines are indented further and never match). +_DATA_KEY_RE = re.compile(r'^ ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')') + + +def cluster_vars_keys(path: Path) -> set[str]: + keys: set[str] = set() + in_data = False + for raw in path.read_text().splitlines(): + if re.match(r"^data:\s*$", raw): + in_data = True + continue + if not in_data: + continue + if st._TOP_KEY_RE.match(raw): # a new top-level key ends the data block + break + m = _DATA_KEY_RE.match(raw) + if m: + keys.add(m.group(1)) + return keys + + +VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml" +KEYS = cluster_vars_keys(VARS_FILE) +check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed") + +# ------------------------------------------- Flux Kustomizations with substituteFrom +# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc +# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND +# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the +# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped. +_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M) +_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)") +_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)") + + +def substitute_sources(doc: str) -> list[tuple[str, str]]: + """[(kind, name)] declared under postBuild.substituteFrom in this doc.""" + m = _SUBFROM_RE.search(doc) + if not m: + return [] + indent = len(m.group(1)) + out: list[tuple[str, str]] = [] + kind = "" + for raw in doc[m.end():].splitlines(): + if raw.strip() == "" or raw.lstrip().startswith("#"): + continue + if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"): + break # dedented out of the list + km = _SUBITEM_KIND_RE.match(raw) + if km: + kind = km.group(1) + continue + nm = _SUBITEM_NAME_RE.match(raw) + if nm and kind: + out.append((kind, nm.group(1))) + kind = "" + return out + + +kustomizations: list[dict] = [] +for yfile in sorted((ROOT / "clusters").rglob("*.yaml")): + text = yfile.read_text() + if st.FLUX_API not in text: + continue + for doc in st._DOC_SPLIT_RE.split(text): + if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M): + continue + pm = st._PATH_RE.search(doc) + if not pm: + continue + nm = st._NAME_RE.search(doc) + srcs = substitute_sources(doc) + if not srcs: + continue # no substitution => nothing to audit + kustomizations.append({ + "file": str(yfile.relative_to(ROOT)), + "name": nm.group(1) if nm else "?", + "path": pm.group(1), + "sources": srcs, + }) + +check(len(kustomizations) > 0, + "no Flux Kustomization with postBuild.substituteFrom found under clusters/ — " + "the parser or the repo layout changed; this check would silently audit nothing") + +# ------------------------------------------------------------------ token scanning +# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree: +# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only +# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders). +TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}") +NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES +# these, so an undefined name yields the default/empty AT RECONCILE TIME. +OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$") +LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$") + + +def classify(dollars: int, inner: str) -> tuple[str, str] | None: + """Return (severity-tag, remediation) for a hazardous expansion, else None.""" + if dollars >= 2: + return None # escaped: emitted as a literal + if NAME_RE.match(inner): + if inner in KEYS: + return None # defined knob — the happy path + return ("UNDEFINED", + f"strict mode fails the Kustomization; today it renders EMPTY. Either add " + f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or " + f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer") + m = OP_RE.match(inner) or LEN_RE.match(inner) + if m: + base = m.group(1) + if base in KEYS: + return None # defined knob with an operator + return ("SILENT-EMPTY", + f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a " + f"cluster-var, so the rendered manifest carries the default, not the " + f"expression. StrictPostBuildSubstitutions does NOT catch this form. " + f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it") + return ("FATAL-PARSE", + "envsubst cannot parse this and fails the ENTIRE Kustomization (every resource " + "in the path). Escape it as $${...} or rewrite the prose without braces") + + +def rendered_files(path_spec: str) -> tuple[list[str], bool]: + """Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively.""" + rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "." + if not (ROOT / rel).is_dir(): + return [], False + applied, _ = st.reconciled_files(ROOT, rel) + return sorted(applied), True + + +findings: list[str] = [] +referenced: dict[str, set[str]] = {} +inventory: list[str] = [] + +for k in kustomizations: + files, exists = rendered_files(k["path"]) + check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' " + f"which is not a directory in this repo") + has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"] + n_def = n_esc = n_bad = 0 + for rel in files: + f = ROOT / rel + try: + text = f.read_text() + except (OSError, UnicodeDecodeError): + continue + for lineno, line in enumerate(text.splitlines(), 1): + for m in TOKEN_RE.finditer(line): + dollars, inner = len(m.group(1)), m.group(2) + verdict = classify(dollars, inner) + if verdict is None: + if dollars >= 2: + n_esc += 1 + else: + n_def += 1 + referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner, + set()).add(k["name"]) + # (4) a DEFINED knob is only substituted if this Kustomization + # actually pulls cluster-vars in; otherwise the literal ships. + check(has_cluster_vars, + f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but " + f"Kustomization '{k['name']}' ({k['file']}) does not " + f"substituteFrom ConfigMap/cluster-vars — the literal token " + f"would ship into the cluster") + continue + n_bad += 1 + tag, fix = verdict + findings.append(f" [{tag}] {rel}:{lineno} {m.group(0)}\n" + f" rendered by Flux Kustomization '{k['name']}' " + f"({k['file']}, path {k['path']})\n" + f" {fix}\n" + f" | {line.strip()[:120]}") + inventory.append(f" {k['name']:<16} path={k['path']:<22} files={len(files):<3} " + f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}") + +check(not findings, + f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths") + +# --------------------------------------------- (5) overlay knobs exist in the base +# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` / +# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml +# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or +# manifest references it — that ordering is what closes the define-then-use race. +_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M) +for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + for key in set(_OP_PATH_RE.findall(overlay.read_text())): + check(key in KEYS, + f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE " + f"default in clusters/base/cluster-vars.yaml — a second overlay would render " + f"${{{key}}} empty (R-ENV-02: author once, supply per env)") + +# ------------------------------------------------------- (6) the check is load-bearing +L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" +GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" +check("flux-postbuild-substitution/check.sh" in L0, + "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped " + "conformance selection can filter this check out of a gate run, so l0.sh must invoke " + "it directly and unconditionally or the gate is not actually enforcing it") +check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# ------------------------------------------------------------- advisory: dead knobs +for key in sorted(KEYS - set(referenced)): + ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest " + f"(dead knob, or documentation-only — advisory, not a failure)") + +# ------------------------------------------------------------------------- report +print("Flux postBuild substitution inventory (per Kustomization):") +for line in inventory: + print(line) +print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}") + +if ADVISORIES: + print("\nADVISORIES (not failures):") + for a in ADVISORIES: + print(f" - {a}") + +if findings: + print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr) + for f in findings: + print(f, file=sys.stderr) + +print() +if not FAILURES: + print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions " + f"(R-ENV-02, R-GITOPS-01)") + raise SystemExit(0) +print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, " + f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) +for f in FAILURES: + print(f" FAIL {f}", file=sys.stderr) +raise SystemExit(1) +PY diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index fa959a9..55202bf 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -311,6 +311,32 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be own comment flags this as owed "when hardware is built." - [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`. +- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing + the `dependsOn` fix. Only the first is now closed: + - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became + *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries + `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block + forever on a dependency that does not exist and nothing downstream would reconcile. + - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key + as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev + exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster + (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only. + - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision + 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and + will be decommissioned once prod is stable. Recorded consequence: once prod is live, + **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window. + The URL is patched in but **cannot work on a cold prod cluster**, for two independent + reasons found 2026-09-01: + 1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for + decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod + loses its source of truth. **This is the decommission plan's first task, not its last.** + 2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire + `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`: + prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve + `gitea.home.keithjackson.dev` to itself and never reach a repository. + Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a + cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own + Phase-6 cutover. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03). diff --git a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 index 0704adb..8a22edf 100644 --- a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 +++ b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 @@ -42,5 +42,22 @@ cgroup: enabled: false hostRoot: /run/cilium/cgroupv2 +# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the +# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a +# 3-node cluster lets one bad render take down two of three agents simultaneously — +# exactly the 2026-09-01 incident, where k8sServiceHost: "" made the agent's `config` +# init container fall back to the in-cluster ClusterIP 10.43.0.1:443. Only the Cilium +# agent can translate that ClusterIP, and the agent in question was the one being +# replaced: `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked. +# A CNI rollout has no healthy-peer fallback — the component being upgraded is the one +# the upgrade needs to reach the API server. This bites hardest on a COLD BOOTSTRAP +# (this role's job), where nothing healthy exists to recover from. With maxUnavailable: 1 +# the DaemonSet will not proceed past an unready pod, bounding damage to a single node. +# Cost: Cilium upgrades serialise and take longer; at 3 nodes that is an acceptable trade. +updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + operator: replicas: {{ cilium_operator_replicas }} diff --git a/platform/cilium/helmrelease.yaml b/platform/cilium/helmrelease.yaml index 781964d..68086d6 100644 --- a/platform/cilium/helmrelease.yaml +++ b/platform/cilium/helmrelease.yaml @@ -89,6 +89,27 @@ spec: autoMount: enabled: false hostRoot: /run/cilium/cgroupv2 + # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident --- + # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1. + # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate. + # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad + # Helm render take down two of three agents simultaneously. That is exactly what happened + # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config` + # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium + # agent itself can translate — and the agent translating it was the one being replaced. + # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once. + # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the + # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet + # will not proceed past an unready pod, so a bad render costs one node, not the cluster. + # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is + # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH). + # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and + # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster + # grows enough that serialised CNI upgrades become an operational problem. + updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field # (Flux/prod may differ). Only the [parity] values above must match the bootstrap. diff --git a/platform/cilium/values.yaml b/platform/cilium/values.yaml index 3223eaf..e9f791b 100644 --- a/platform/cilium/values.yaml +++ b/platform/cilium/values.yaml @@ -42,6 +42,16 @@ cgroup: autoMount: enabled: false hostRoot: /run/cilium/cgroupv2 +# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart +# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the +# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the +# ClusterIP 10.43.0.1:443, translatable only by the very agent being replaced). A CNI upgrade +# has no healthy-peer fallback, least of all on a cold prod bootstrap; maxUnavailable: 1 stops +# the rollout at the first unready pod. Cost: serialised, slower upgrades (fine at 3 nodes). +updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 # NON-parity (Flux/prod may differ; dev sets 1 to save RAM) — omitted from the HelmRelease: # operator.replicas # diff --git a/platform/longhorn/helmrelease.yaml b/platform/longhorn/helmrelease.yaml index a434558..94d9c07 100644 --- a/platform/longhorn/helmrelease.yaml +++ b/platform/longhorn/helmrelease.yaml @@ -2,7 +2,7 @@ # Longhorn — distributed replicated block storage across the 3 nodes (ADR-0003). # Serves R-STATE-01 (state survives destroy/rebuild) and R-STATE-02 (3-way replication). # -# Dev-appropriate values. Environment differences enter ONLY via ${...} substitution from +# Dev-appropriate values. Environment differences enter ONLY via $${...} substitution from # cluster-vars (R-ENV-02); everything else is identical base config for dev and prod. apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease diff --git a/platform/longhorn/storageclasses.yaml b/platform/longhorn/storageclasses.yaml index 5f510b3..925ddac 100644 --- a/platform/longhorn/storageclasses.yaml +++ b/platform/longhorn/storageclasses.yaml @@ -20,7 +20,7 @@ parameters: # (replicaSoftAntiAffinity=false) guarantees these land on distinct nodes, so any single node # loss keeps the volume available. # HARDCODED, not ${LONGHORN_DEFAULT_REPLICA_COUNT}: StorageClass.parameters is map[string]string, - # but kustomize strips the quotes around a bare "${VAR}" placeholder, so Flux postBuild substitution + # but kustomize strips the quotes around a bare "$${VAR}" placeholder, so Flux postBuild substitution # yields an unquoted numeric (3) that server-side apply rejects (int, not string). A quoted numeric # literal like "3"/"30" IS preserved by kustomize. The replica knob still drives the GLOBAL default # via defaultSettings.defaultReplicaCount in helmrelease.yaml (a Helm int value, unaffected).