From b8602c939cf82cbcb0e55275a7463636948d3555 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 01:00:16 -0500 Subject: [PATCH 1/7] =?UTF-8?q?docs:=20INCIDENT=20=E2=80=94=20Flux=20post-?= =?UTF-8?q?build=20substitution=20fails=20OPEN,=20briefly=20broke=20Cilium?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Merging PR #4 put cilium agents into Init:Error on 2/3 dev nodes for ~6 minutes. Self-healed; cluster fully green. Recorded because the mechanism is systemic and far more dangerous on a cold prod bootstrap. Root cause: cluster-vars (which DEFINES a knob) is owned by the flux-system Kustomization, while platform/ (which REFERENCES it) is owned by the platform Kustomization, with no dependency ordering between defining a variable and using it. Flux post-build substitution is NON-STRICT by default, so during the window where platform rendered the new manifest against the old cluster-vars, ${K3S_API_HOST} became the EMPTY STRING. Cilium upgraded with k8sServiceHost: "" and fell back to ClusterIP 10.43.0.1 — which only the Cilium agent can translate, and that agent was the thing restarting. Its config init container then could not reach the API server at all. Proven by the HelmRelease config digest going e83bf695 -> eebd6c88 -> e83bf695: the config changed and then changed back to BYTE-IDENTICAL. The pre-merge byte-identity verification was correct — the steady state is identical; the damage was a transient intermediate render. optional: false does NOT protect against this: it requires the ConfigMap to exist, not that every referenced key is defined. Fix recommended, NOT applied unilaterally: enable the StrictPostBuildSubstitutions feature gate on kustomize-controller so an undefined variable FAILS the Kustomization instead of silently emptying it — fail-closed, per this project's own doctrine. Requires first proving every ${VAR} in every substituted path is defined or correctly escaped, or it would fail legitimate embedded shell expansions. Interim discipline: add a knob in one commit, reference it in a later one. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01. --- DECISIONS-LOG.md | 86 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 63b347c..1e97184 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1496,3 +1496,89 @@ thing that ever closes a requirement. chart default rather than set explicitly — see the follow-up in `platform/longhorn/`. - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI. + +--- + +## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.** + +Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put +`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes** +and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down). +Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold +prod bootstrap than it was here. + +### What happened + +The HelmRelease config digest tells the whole story: + +``` +v8 05:49:39 digest e83bf695… <- before +v9 05:50:11 digest eebd6c88… <- the bad render +v10 05:55:23 digest e83bf695… <- identical to v8 again +``` + +The config changed and then changed **back to byte-identical**. Nothing in git differed +semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus +`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets +`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady +state is identical. The damage happened in a transient intermediate state. + +### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations + +- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization. +- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`** + Kustomization, via `postBuild.substituteFrom: cluster-vars`. +- There is **no dependency ordering between defining a variable and using it.** + +So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`, +which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an +undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm +upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` — +which only the Cilium agent itself can translate. The agent was the thing restarting, so its own +`config` init container could not reach the API server: + +``` +level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443 + error="dial tcp 10.43.0.1:443: i/o timeout" +level=fatal msg="Build config failed" +``` + +Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the +correct value, digest returned to `e83bf695…`, and all three agents came up. + +`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires +the ConfigMap to *exist*, not that every referenced key is defined. + +### Why this matters much more for prod + +On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the +race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**, +there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may +never come up at all — with no healthy peer to recover from. This is a plausible +bring-up-blocker for `clusters/prod`. + +### The fix — make substitution FAIL CLOSED + +`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict +substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail +the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine +("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts +this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile +failure*. + +**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization +that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in +embedded scripts. The safe sequence, before flipping it: +1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in + `cluster-vars` or correctly escaped as `$${VAR}`. +2. Then enable the gate, and let CI + a dev reconcile prove it. + +Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and +reference it in a later one**, so the definition is always reconciled before its first use. + +### Operational note worth keeping + +A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the +`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is +set, and the agent that translates that ClusterIP is the one being replaced. Keep +`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time. From 34849559032da8b2c31ae16ba14333647934e030 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 01:29:40 -0500 Subject: [PATCH 2/7] =?UTF-8?q?fix(sso):=20gitea=20+=20nextcloud=20OIDC=20?= =?UTF-8?q?bootstrap=20was=20DEAD=20CODE=20=E2=80=94=20envsubst=20ate=20th?= =?UTF-8?q?e=20guard?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED braced expansion: if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0 apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long before the container ever ran. Verified in the LIVE cluster: the deployed Deployments literally contain if [ -z "" ]; then which is always true. The entire OIDC registration block has been skipped on EVERY boot of both apps. WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list` shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider. Those live in each app's own database, persisted from an earlier successful registration. SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO on gitea or nextcloud. The irony: the surrounding code was hardened twice — see its own comments about a "silent first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was unreachable, because the guard above it always evaluated true. Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871). FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated. Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst the container now receives the intended `${VAR:-}` instead of `""`. STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they "intentionally handle the unset case". So a targeted check was required. MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section wired into emit(), so every render path refuses to emit a manifest containing a braced expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not), and indexed forms get their own hard-error message. Nine new --selftest assertions, including the exact regression. Proven to bite: rendering the pre-fix catalog files from HEAD now exits 1 with the precise diagnosis. Sweep: the complete substituted surface was enumerated from every Kustomization's path. catalog/ contains exactly TWO braced expansions and both were these bugs; every other braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero undeclared names); existing $${UOS_URL} escapes verified intact. Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for all 10 onboarded services; the three L1 render checks pass. NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll — strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The OIDC block then executes for the first time in these apps' deployed lifetime. Both paths are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's, this is when it gets corrected instead of silently left stale. Tail both pods after merge. Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/gitea.yaml | 4 +- apps/wave-1/nextcloud.yaml | 5 +- catalog/README.md | 10 ++ catalog/gitea.yaml | 14 ++- catalog/nextcloud.yaml | 14 ++- catalog/render.py | 221 ++++++++++++++++++++++++++++++++++++- 6 files changed, 258 insertions(+), 10 deletions(-) diff --git a/apps/wave-0/gitea.yaml b/apps/wave-0/gitea.yaml index 067949e..359b9df 100644 --- a/apps/wave-0/gitea.yaml +++ b/apps/wave-0/gitea.yaml @@ -127,8 +127,8 @@ spec: command: - /bin/sh - -c - - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"$GITEA_ADMIN_USERNAME@home.local\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name\ - \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n" + - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"$GITEA_ADMIN_USERNAME@home.local\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\ + \ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n" volumes: - name: data persistentVolumeClaim: diff --git a/apps/wave-1/nextcloud.yaml b/apps/wave-1/nextcloud.yaml index 6f68a40..b1a0f3d 100644 --- a/apps/wave-1/nextcloud.yaml +++ b/apps/wave-1/nextcloud.yaml @@ -125,8 +125,9 @@ spec: command: - /bin/sh - -c - - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra\ - \ positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# ` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n" + - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"\ + [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# ` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"\ + $ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n" volumes: - name: data persistentVolumeClaim: diff --git a/catalog/README.md b/catalog/README.md index 4b5912a..271de76 100644 --- a/catalog/README.md +++ b/catalog/README.md @@ -28,6 +28,16 @@ inherit every guarantee automatically. `auth.groups`; `native`/`none` are documented exceptions only. - Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`. +- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a + `lifecycle` postStart script, an `env` value — lands under `apps/wave-/`, and every + `apps-wave-` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs + envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the + *shell* is replaced with the empty string before the container ever starts (this silently killed + gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and + need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and + written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml). + `render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode + cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms). ## Validate before commit — single command, no cluster (R-TEST-02) ```bash diff --git a/catalog/gitea.yaml b/catalog/gitea.yaml index ab07fba..f7a2885 100644 --- a/catalog/gitea.yaml +++ b/catalog/gitea.yaml @@ -109,7 +109,19 @@ lifecycle: echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2 { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true } - if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then + # NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild. + # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE + # time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not + # a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE + # — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on + # every boot since 8738871 (2026-08-27). SSO kept working only because the login source was + # already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all. + # Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design. + # `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell. + # Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the + # `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke. + # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py. + if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)" else # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200). diff --git a/catalog/nextcloud.yaml b/catalog/nextcloud.yaml index f28aba1..3723d89 100644 --- a/catalog/nextcloud.yaml +++ b/catalog/nextcloud.yaml @@ -111,7 +111,19 @@ lifecycle: # every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27. OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \ || echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2 - if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then + # NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild. + # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE + # time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is + # not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS + # TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never + # installing/enabling user_oidc nor registering the provider. SSO kept working only because + # the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with + # NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator + # forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion + # for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why + # `--clientsecret-env` and the bare uses below always worked and only the guard broke. + # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py. + if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)" exit 0 fi diff --git a/catalog/render.py b/catalog/render.py index 1873554..992f531 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -29,6 +29,7 @@ conformance harness stays stdlib-only and only shells out to this script + diffs from __future__ import annotations import argparse +import re import sys from pathlib import Path @@ -936,6 +937,124 @@ def r_virtualmachine(s: dict) -> dict: } +# ============================================================================= +# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that +# Flux's envsubst would silently eat before the workload ever runs. +# ============================================================================= +# Every manifest this renderer writes lands under apps/wave-/, and EVERY apps-wave- +# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see +# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered +# document — including any shell script carried verbatim in a `lifecycle` hook or an env +# value — at reconcile time, long before a container sees it. The escape contract is stated +# in clusters/base/apps-vms.yaml and proven with `flux envsubst`: +# +# `${VAR}` -> SUBSTITUTED (empty when unset) => a shell brace must be written `$${VAR}`, +# which envsubst emits back as `${VAR}`. +# `$VAR` -> left alone => bare shell vars need NO escaping. +# `$(cmd)` -> left alone => command substitution needs NO escaping. +# `${A[0]}` -> HARD ERROR ("missing closing brace"), failing the whole Kustomization. +# +# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC +# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var, +# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the +# registration entirely. Flux's STRICT substitution mode cannot catch this class: it +# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is +# exactly the form a defensive shell script uses. So the renderer checks it itself. +CLUSTER_VARS_PATH = ( + Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml" +) + +# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N +# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is +# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms +# (`${A[0]}`) are caught too rather than skipped. +_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}") +_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z") + +_CLUSTER_VARS_CACHE: dict = {} + + +def cluster_vars_keys(path: Path | None = None) -> set: + """The declared cluster-vars keys — the ONLY names envsubst may legitimately expand. + + Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the + renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e. + FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell. + """ + key = str(path or CLUSTER_VARS_PATH) + if key not in _CLUSTER_VARS_CACHE: + try: + doc = yaml.safe_load(Path(key).read_text()) + data = doc.get("data") if isinstance(doc, dict) else None + _CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set() + except Exception: # pragma: no cover - absent/malformed => fail closed + _CLUSTER_VARS_CACHE[key] = set() + return _CLUSTER_VARS_CACHE[key] + + +def substitution_violations(text: str, known: set | None = None) -> list: + """Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key. + + Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only + when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug: + a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that + envsubst will blank, or an indexed form that hard-fails the whole Kustomization. + """ + keys = cluster_vars_keys() if known is None else known + out = [] + for lineno, line in enumerate(text.splitlines(), start=1): + for m in _BRACED_EXPANSION_RE.finditer(line): + if len(m.group(1)) % 2 == 0: + continue # even run of `$` => escaped; envsubst emits it back literally + expr = m.group(2) + if _PLAIN_IDENT_RE.match(expr): + if expr in keys: + continue # a declared cluster-vars knob, intentionally substituted + reason = ( + f"'{expr}' is not a declared cluster-vars key — envsubst will replace " + "it with the EMPTY STRING" + ) + elif "[" in expr: + reason = ( + "indexed expansion — envsubst hard-fails with 'missing closing brace' " + "and the whole Kustomization stops reconciling" + ) + else: + reason = ( + "shell expansion (default-operator/parameter form) — envsubst eats it " + "at reconcile time and STRICT mode exempts this form, so nothing else " + "will catch it" + ) + out.append((lineno, m.group(0), reason)) + return out + + +def assert_substitution_safe(text: str, origin: str = "") -> None: + """Refuse to emit a manifest carrying an unescaped braced expansion (see above). + + Raises ValueError listing every offending token. The fix is almost always to escape the + shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the + value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml + and write it as a bare `${KNOB}`. + """ + bad = substitution_violations(text) + if not bad: + return + lines = [ + f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's " + "postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:" + ] + for lineno, token, reason in bad: + lines.append(f" line {lineno}: {token} -> {reason}") + lines.append( + " FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal " + "`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob " + "must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. " + "Contract: clusters/base/apps-vms.yaml." + ) + raise ValueError("\n".join(lines)) + + # ============================================================================= # Top-level render + emit # ============================================================================= @@ -1026,8 +1145,14 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None) return docs -def emit(docs: list) -> str: - """Deterministic multi-document YAML text (golden-diffable).""" +def emit(docs: list, origin: str = "") -> str: + """Deterministic multi-document YAML text (golden-diffable). + + Every emitted document is checked against the substitution contract before it is handed + back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs + envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises + ValueError rather than emitting such a manifest. + """ parts = [] for comment, resource in docs: body = yaml.dump( @@ -1037,7 +1162,9 @@ def emit(docs: list) -> str: width=4096, ) parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n") - return "".join(parts) + text = "".join(parts) + assert_substitution_safe(text, origin) + return text def load_sd(path: Path) -> dict: @@ -1076,7 +1203,9 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N src_dir = catalog_dir if catalog_dir is not None else path.parent rev = build_reverse_map(_load_catalog(src_dir)) inbound = rev.get(sd["name"], []) - return emit(render(sd, image_automation=image_automation, inbound=inbound)) + return emit( + render(sd, image_automation=image_automation, inbound=inbound), origin=str(path) + ) # ============================================================================= @@ -1729,6 +1858,90 @@ def _selftest() -> int: check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"], "wave kustomization must list exactly the wave's resources") + # 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose + # shell script carries an unescaped `${...}`. Every rendered manifest lands under a + # Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before + # the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]` + # -> `[ -z "" ]`, always true, whole registration block skipped on every boot). + known = {"BASE_DOMAIN", "GATEWAY_VIP"} + # allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution. + check( + substitution_violations("host: app.${BASE_DOMAIN}", known) == [], + "a bare declared cluster-vars key must be allowed (that is what substitution is FOR)", + ) + # allowed: escaped shell braces, and the forms envsubst leaves alone. + check( + substitution_violations('if [ -z "$${X:-}" ]; then', known) == [], + "an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)", + ) + check( + substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [], + "bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged", + ) + # `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live. + check( + len(substitution_violations("$$${X:-}", known)) == 1, + "an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)", + ) + # flagged: the exact regression — a default-operator form STRICT mode deliberately exempts. + guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known) + check( + len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}", + "an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)", + ) + # flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error). + check( + len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1, + "an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)", + ) + idx = substitution_violations("echo ${PIPESTATUS[0]}", known) + check( + len(idx) == 1 and "missing closing brace" in idx[0][2], + "an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)", + ) + # the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the + # bug RAISES rather than writing a manifest whose guard is dead on arrival. + bad_sd = { + "name": "subbad", + "source": {"image": "a:1"}, + "lifecycle": { + "postStart": { + "exec": { + "command": [ + "/bin/sh", + "-c", + 'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi', + ] + } + } + }, + } + try: + emit(render(bad_sd), origin="") + check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion") + except ValueError as exc: + check( + "SUBBAD_SECRET" in str(exc) and "$${" in str(exc), + "the refusal must name the offending token AND state the `$${...}` fix", + ) + # the ESCAPED form of the same hook renders normally (the fix must be emittable). + good_sd = dict(bad_sd, name="subgood") + good_sd["lifecycle"] = { + "postStart": { + "exec": { + "command": [ + "/bin/sh", + "-c", + 'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi', + ] + } + } + } + check( + "$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin=""), + "the escaped `$${...}` form must render (and reach the shell as `${...}`)", + ) + if failures: for f in failures: sys.stderr.write(f"SELFTEST FAIL: {f}\n") From 8a2be824f600e6baaeb174d9aacc2605fee27923 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 01:31:07 -0500 Subject: [PATCH 3/7] feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a path renders" has exactly one definition. It asserts: every ${...} on a substituted path is defined-or-escaped (comments included); no default-operator form on an undefined name; no unparseable expansion; the rendering path actually substitutes cluster-vars; every overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing. WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 — so an edit to a substituted manifest under apps/ or platform/ would have filtered this check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh. Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form, missing substituteFrom, un-wired gate); each fails as intended, baseline green. STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone | flux envsubst --strict` exits 0 against cluster-vars base defaults. Negative control confirms the incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped here — that is a live-cluster change and the operator's call. Escapes the six comment hazards that stood in the way (apps/vms/unifi x4, platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own contract is that $${...} emits a literal ${...}, escaping them is both precise and self-consistent. kustomize strips these particular comments today, but comment survival is an implementation detail — a body comment carrying $ WAS observed surviving into built apps/wave-1 — and a survivor would kill the whole Kustomization. DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place. Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the trigger for removing them. Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV, CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames are hardcoded elsewhere) and are worth a follow-up. Gate: MERGE GATE: PASS. 102 assertions, 0 hazards. Serves: R-ENV-02, R-GITOPS-01. Co-Authored-By: Claude Opus 5 (1M context) --- apps/vms/unifi/30-virtualmachine.yaml | 11 +- apps/vms/unifi/50-service-l4.yaml | 2 +- ci/l0.sh | 37 ++ conformance/run.sh | 28 +- .../chainsaw-test.yaml | 60 +++ .../flux-postbuild-substitution/check.sh | 369 ++++++++++++++++++ platform/longhorn/helmrelease.yaml | 2 +- platform/longhorn/storageclasses.yaml | 2 +- 8 files changed, 491 insertions(+), 20 deletions(-) create mode 100644 conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/flux-postbuild-substitution/check.sh diff --git a/apps/vms/unifi/30-virtualmachine.yaml b/apps/vms/unifi/30-virtualmachine.yaml index dc9a963..ae0d159 100644 --- a/apps/vms/unifi/30-virtualmachine.yaml +++ b/apps/vms/unifi/30-virtualmachine.yaml @@ -92,10 +92,13 @@ spec: # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData. # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits - # as a literal ${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone - # by envsubst and need no escaping. NEVER write an unescaped ${...} here, and never write - # ${ARR[0]} even escaped-adjacent: envsubst hard-fails on `[` ("missing closing brace") - # and would break the whole apps-vms apply. + # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone + # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write + # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace") + # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against + # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal + # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not" + # wording in clusters/base/apps-vms.yaml overstated it. userData: |- #cloud-config hostname: unifi diff --git a/apps/vms/unifi/50-service-l4.yaml b/apps/vms/unifi/50-service-l4.yaml index 4ba925a..a380ee6 100644 --- a/apps/vms/unifi/50-service-l4.yaml +++ b/apps/vms/unifi/50-service-l4.yaml @@ -27,7 +27,7 @@ # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53). # Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for -# exactly this token; every shell ${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...} +# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...} # accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool. # # REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an diff --git a/ci/l0.sh b/ci/l0.sh index d4dc4cc..efcedda 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -100,6 +100,43 @@ else fi fi +# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ---------- +# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it +# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still +# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes +# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every +# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED +# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail +# a legitimate manifest, and that no expansion silently empties today. +# +# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the +# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys +# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/ +# attributes no requirement, so the requirement-scoped selection would filter this check +# out of exactly the change that introduces the bug. The invariant is whole-repo (same +# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation. +# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a +# sub-second offline scan with no side effects, so the duplication is not worth avoiding. +SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}" +log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})" +if [ ! -f "$SUBST_CHECK" ]; then + skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green" + summary "- subst-safety: **SKIPPED** (check missing)" + if [ "$rc" -eq 0 ]; then rc=3; fi +else + subst_rc=0 + bash "$SUBST_CHECK" || subst_rc=$? + case "$subst_rc" in + 0) ok "Flux postBuild substitution safety passed" + summary "- subst-safety: **passed**" ;; + 3) skip "substitution check SKIPPED (clusters/ absent) — not a pass" + summary "- subst-safety: **SKIPPED** (nothing to audit)" + if [ "$rc" -eq 0 ]; then rc=3; fi ;; + *) err "Flux postBuild substitution safety FAILED" + summary "- subst-safety: **FAILED**"; rc=1 ;; + esac +fi + case "$rc" in 0) ok "L0 stage complete" ;; 3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;; diff --git a/conformance/run.sh b/conformance/run.sh index 2d9efcc..6321768 100755 --- a/conformance/run.sh +++ b/conformance/run.sh @@ -57,17 +57,15 @@ FORCE_NO_CLUSTER=0 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the # exemption is explicit and reviewable, not silent. -# R-ENV-02 (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in -# coverage.md). The coarse path->requirement map keys -# clusters/* AND ci/* to it and the R-ENV-02 doc token -# appears in platform/kustomization.yaml's header, so ANY -# infra/overlay/CI edit (e.g. enabling a component, bumping -# SPEC_PIN) would otherwise false-block on it. -# R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/* -# path-map keys to it, so editing any ci/ file (SPEC_PIN -# adoption included) would false-block. -# These are visible GAPs, not hidden — removing them from this list is the right move the -# moment WS-TEST adds their verifying tests. +# +# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted +# because they were unscaffolded while the coarse path->requirement map keys clusters/* +# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real +# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the +# substitution mechanism that IS the "authored once, promoted" contract, so they are +# covered rather than exempt and the completeness gate can hold them to it. That test +# names its own limits in its header: it proves the STATIC property (no manifest renders +# an undefined variable), not the live-cluster half of either requirement. # # FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header # tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in @@ -83,7 +81,7 @@ FORCE_NO_CLUSTER=0 # R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra") # but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no # longer a gap and has been removed from this list; the coverage report counts it as covered. -DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01" +DOCUMENTED_GAPS="" usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } @@ -108,7 +106,11 @@ log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; } # --- Coverage report is always produced (cluster-independent, R-TEST-04) -------------- gen_coverage() { - local flags=(--allow-gap "${DOCUMENTED_GAPS}") + # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no + # requirement is exempt, and handing the flag an empty string would be a silent no-op + # that reads like an exemption in the CLI log. + local flags=() + [ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}") [ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict) # A tagged subset run is also the merge gate: the touched requirements MUST be covered. [ -n "${REQS}" ] && flags+=(--require "${REQS}") diff --git a/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml b/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml new file mode 100644 index 0000000..b048116 --- /dev/null +++ b/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml @@ -0,0 +1,60 @@ +# ============================================================================= +# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism: +# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each +# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes +# the empty string while the Kustomization still reports Ready — which emptied Cilium's +# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01). +# +# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom, +# WHEN each `spec.path` is resolved to the exact file set it renders (following +# kustomization.yaml `resources:` transitively, not globbing the repo), +# THEN every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml +# or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default +# operator on an undefined name (the silent-empty class strict mode does NOT catch); +# no unescaped expansion is unparseable by envsubst (which fails the WHOLE +# Kustomization); every path rendering a knob actually substitutes cluster-vars; +# every overlay-patched knob has a base default; and the check is wired into the +# merge gate as a blocking, unconditionally-run stage. +# +# This is the prerequisite proof for enabling kustomize-controller's +# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw +# Test wraps the SAME script. +# +# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests +# a path includes — a deliberate superset of the post-`kustomize build` stream. It does +# not render, does not prove the feature gate is enabled on the live kustomize-controller, +# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice +# only: what git says a Kustomization renders is fully determined, with no variable +# silently emptied on the way in. The live-state half is L2, untested today. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: flux-postbuild-substitution + labels: + conformance.home.k3s/R-ENV-02: "true" + conformance.home.k3s/R-GITOPS-01: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every variable a substituted Flux path renders is defined in cluster-vars or + escaped, so post-build substitution cannot silently empty a value and enabling + StrictPostBuildSubstitutions cannot fail a legitimate manifest. +spec: + steps: + - name: audit-every-substituted-flux-path + description: >- + Resolve each postBuild.substituteFrom Kustomization to the files it renders and + assert every ${...} is defined or escaped, then assert the gate wires it blocking. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/flux-postbuild-substitution/check.sh b/conformance/tests/l0-static/flux-postbuild-substitution/check.sh new file mode 100755 index 0000000..c05c725 --- /dev/null +++ b/conformance/tests/l0-static/flux-postbuild-substitution/check.sh @@ -0,0 +1,369 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — flux-postbuild-substitution +# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)." +# — spec docs/02-environments-and-provisioning.md) +# R-GITOPS-01 ("All desired state in git, reconciled by Flux." +# — spec docs/05-gitops-and-delivery.md) +# +# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN +# -------------------------------------------------------- +# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml` +# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders, +# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an +# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports +# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put +# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that +# fails OPEN"). "Authored once, promoted" is therefore only true while every token a +# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in +# the repo asserted that. This check does. +# +# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain, +# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption): +# ${DEFINED} -> substituted with the cluster-vars value. +# ${UNDEFINED} -> EMPTY STRING, silently. `--strict` / the cluster-wide +# StrictPostBuildSubstitutions feature gate turns this into a +# loud Kustomization failure ("variable not set (strict mode)"). +# ${UNDEFINED:-d} -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does +# NOT catch this (it exempts default-operator forms by design), +# so a shell guard written `[ -z "${X:-}" ]` inside a rendered +# manifest is pre-resolved to `[ -z "" ]` and its else-branch +# becomes unreachable in the container. Silent, and permanent. +# $${VAR}, $${V:-d}, $${ARR[0]} -> emitted as the LITERAL ${...} for a downstream +# shell/cloud-init consumer. Escaping is total: it neutralises +# even the indexed form. +# $VAR, $(cmd) -> left alone by envsubst; they need no escaping. +# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse +# variable name" / "missing closing brace") which fails the +# WHOLE Kustomization — every resource in the path, not just +# the offending one. +# The escape contract this enforces is the one already documented in +# `clusters/base/apps-vms.yaml`; this check is that comment made executable. +# +# GIVEN the repository as authored, +# WHEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom +# is resolved to the exact file set its `spec.path` renders (by following +# kustomization.yaml `resources:` transitively — a path, not a glob), +# THEN (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`; +# (2) no UNESCAPED expansion carries a default/length/substitution operator on an +# undefined name (the silent-empty class strict mode does not catch); +# (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) — +# that fails the entire Kustomization, so it is fatal wherever it appears; +# (4) every Kustomization that renders a cluster-vars token actually declares +# `substituteFrom` a ConfigMap named cluster-vars (else the literal token +# ships into the cluster); +# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap — +# the "add the knob with a base default BEFORE referencing it" rule that the +# incident's interim discipline depends on; +# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and +# ci/gate.sh sets the merge block when the l0 stage fails. +# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build` +# preserves body comments into the stream envsubst reads (verified: a `$`-carrying +# comment survives into the built apps/wave-1 output), so a braced expression in a +# comment is a live hazard, not prose. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check): +# - It does NOT render the manifests. It scans the SOURCE files a path includes, +# which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize +# drops some comments). A finding in a comment may not reach envsubst TODAY; it is +# still reported, because whether a given comment survives the build is a kustomize +# implementation detail that changes with the pinned version, and the failure it +# causes is total (assertion 3). The end-to-end proof — `kustomize build | +# flux envsubst --strict` for every substituted path — needs kustomize + flux, +# which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh +# l0), so it cannot run here without permanently SKIPPING. It was run by hand and +# recorded in DECISIONS-LOG; this check is the reproducible static half. +# - It does NOT prove the cluster's live state matches git, nor that the +# StrictPostBuildSubstitutions feature gate is actually enabled on +# kustomize-controller. Both are cluster facts (L2), untested here. Tagging +# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders +# is fully determined, with no variable silently emptied on the way in. +# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool, +# that an overlay's LAN block is self-consistent). Presence, not correctness. +# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure: +# several are deliberately reference-only and say so in cluster-vars.yaml +# (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so +# failing on them would be a false block on documented intent. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +# Honest SKIP (never green) if the thing under test is not present at all. +if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then + echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2 + exit 3 +fi + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit of Flux postBuild substitution safety. Pure standard library. + +No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are +documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization +`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one +definition of "what a Flux path actually renders" in the harness. +""" +from __future__ import annotations + +import os +import re +import sys +from pathlib import Path + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree +sys.path.insert(0, str(ROOT / "conformance" / "lib")) +import status as st # noqa: E402 — reuse the walk, don't fork it + +FAILURES: list[str] = [] +ADVISORIES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +# ------------------------------------------------------------------ cluster-vars +# Parse contract: `data:` block of a single ConfigMap, one ` KEY: "value"` per line +# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and +# continuation-comment lines are indented further and never match). +_DATA_KEY_RE = re.compile(r'^ ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')') + + +def cluster_vars_keys(path: Path) -> set[str]: + keys: set[str] = set() + in_data = False + for raw in path.read_text().splitlines(): + if re.match(r"^data:\s*$", raw): + in_data = True + continue + if not in_data: + continue + if st._TOP_KEY_RE.match(raw): # a new top-level key ends the data block + break + m = _DATA_KEY_RE.match(raw) + if m: + keys.add(m.group(1)) + return keys + + +VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml" +KEYS = cluster_vars_keys(VARS_FILE) +check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed") + +# ------------------------------------------- Flux Kustomizations with substituteFrom +# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc +# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND +# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the +# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped. +_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M) +_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)") +_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)") + + +def substitute_sources(doc: str) -> list[tuple[str, str]]: + """[(kind, name)] declared under postBuild.substituteFrom in this doc.""" + m = _SUBFROM_RE.search(doc) + if not m: + return [] + indent = len(m.group(1)) + out: list[tuple[str, str]] = [] + kind = "" + for raw in doc[m.end():].splitlines(): + if raw.strip() == "" or raw.lstrip().startswith("#"): + continue + if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"): + break # dedented out of the list + km = _SUBITEM_KIND_RE.match(raw) + if km: + kind = km.group(1) + continue + nm = _SUBITEM_NAME_RE.match(raw) + if nm and kind: + out.append((kind, nm.group(1))) + kind = "" + return out + + +kustomizations: list[dict] = [] +for yfile in sorted((ROOT / "clusters").rglob("*.yaml")): + text = yfile.read_text() + if st.FLUX_API not in text: + continue + for doc in st._DOC_SPLIT_RE.split(text): + if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M): + continue + pm = st._PATH_RE.search(doc) + if not pm: + continue + nm = st._NAME_RE.search(doc) + srcs = substitute_sources(doc) + if not srcs: + continue # no substitution => nothing to audit + kustomizations.append({ + "file": str(yfile.relative_to(ROOT)), + "name": nm.group(1) if nm else "?", + "path": pm.group(1), + "sources": srcs, + }) + +check(len(kustomizations) > 0, + "no Flux Kustomization with postBuild.substituteFrom found under clusters/ — " + "the parser or the repo layout changed; this check would silently audit nothing") + +# ------------------------------------------------------------------ token scanning +# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree: +# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only +# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders). +TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}") +NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES +# these, so an undefined name yields the default/empty AT RECONCILE TIME. +OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$") +LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$") + + +def classify(dollars: int, inner: str) -> tuple[str, str] | None: + """Return (severity-tag, remediation) for a hazardous expansion, else None.""" + if dollars >= 2: + return None # escaped: emitted as a literal + if NAME_RE.match(inner): + if inner in KEYS: + return None # defined knob — the happy path + return ("UNDEFINED", + f"strict mode fails the Kustomization; today it renders EMPTY. Either add " + f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or " + f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer") + m = OP_RE.match(inner) or LEN_RE.match(inner) + if m: + base = m.group(1) + if base in KEYS: + return None # defined knob with an operator + return ("SILENT-EMPTY", + f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a " + f"cluster-var, so the rendered manifest carries the default, not the " + f"expression. StrictPostBuildSubstitutions does NOT catch this form. " + f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it") + return ("FATAL-PARSE", + "envsubst cannot parse this and fails the ENTIRE Kustomization (every resource " + "in the path). Escape it as $${...} or rewrite the prose without braces") + + +def rendered_files(path_spec: str) -> tuple[list[str], bool]: + """Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively.""" + rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "." + if not (ROOT / rel).is_dir(): + return [], False + applied, _ = st.reconciled_files(ROOT, rel) + return sorted(applied), True + + +findings: list[str] = [] +referenced: dict[str, set[str]] = {} +inventory: list[str] = [] + +for k in kustomizations: + files, exists = rendered_files(k["path"]) + check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' " + f"which is not a directory in this repo") + has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"] + n_def = n_esc = n_bad = 0 + for rel in files: + f = ROOT / rel + try: + text = f.read_text() + except (OSError, UnicodeDecodeError): + continue + for lineno, line in enumerate(text.splitlines(), 1): + for m in TOKEN_RE.finditer(line): + dollars, inner = len(m.group(1)), m.group(2) + verdict = classify(dollars, inner) + if verdict is None: + if dollars >= 2: + n_esc += 1 + else: + n_def += 1 + referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner, + set()).add(k["name"]) + # (4) a DEFINED knob is only substituted if this Kustomization + # actually pulls cluster-vars in; otherwise the literal ships. + check(has_cluster_vars, + f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but " + f"Kustomization '{k['name']}' ({k['file']}) does not " + f"substituteFrom ConfigMap/cluster-vars — the literal token " + f"would ship into the cluster") + continue + n_bad += 1 + tag, fix = verdict + findings.append(f" [{tag}] {rel}:{lineno} {m.group(0)}\n" + f" rendered by Flux Kustomization '{k['name']}' " + f"({k['file']}, path {k['path']})\n" + f" {fix}\n" + f" | {line.strip()[:120]}") + inventory.append(f" {k['name']:<16} path={k['path']:<22} files={len(files):<3} " + f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}") + +check(not findings, + f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths") + +# --------------------------------------------- (5) overlay knobs exist in the base +# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` / +# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml +# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or +# manifest references it — that ordering is what closes the define-then-use race. +_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M) +for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + for key in set(_OP_PATH_RE.findall(overlay.read_text())): + check(key in KEYS, + f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE " + f"default in clusters/base/cluster-vars.yaml — a second overlay would render " + f"${{{key}}} empty (R-ENV-02: author once, supply per env)") + +# ------------------------------------------------------- (6) the check is load-bearing +L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" +GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" +check("flux-postbuild-substitution/check.sh" in L0, + "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped " + "conformance selection can filter this check out of a gate run, so l0.sh must invoke " + "it directly and unconditionally or the gate is not actually enforcing it") +check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# ------------------------------------------------------------- advisory: dead knobs +for key in sorted(KEYS - set(referenced)): + ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest " + f"(dead knob, or documentation-only — advisory, not a failure)") + +# ------------------------------------------------------------------------- report +print("Flux postBuild substitution inventory (per Kustomization):") +for line in inventory: + print(line) +print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}") + +if ADVISORIES: + print("\nADVISORIES (not failures):") + for a in ADVISORIES: + print(f" - {a}") + +if findings: + print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr) + for f in findings: + print(f, file=sys.stderr) + +print() +if not FAILURES: + print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions " + f"(R-ENV-02, R-GITOPS-01)") + raise SystemExit(0) +print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, " + f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) +for f in FAILURES: + print(f" FAIL {f}", file=sys.stderr) +raise SystemExit(1) +PY diff --git a/platform/longhorn/helmrelease.yaml b/platform/longhorn/helmrelease.yaml index a434558..94d9c07 100644 --- a/platform/longhorn/helmrelease.yaml +++ b/platform/longhorn/helmrelease.yaml @@ -2,7 +2,7 @@ # Longhorn — distributed replicated block storage across the 3 nodes (ADR-0003). # Serves R-STATE-01 (state survives destroy/rebuild) and R-STATE-02 (3-way replication). # -# Dev-appropriate values. Environment differences enter ONLY via ${...} substitution from +# Dev-appropriate values. Environment differences enter ONLY via $${...} substitution from # cluster-vars (R-ENV-02); everything else is identical base config for dev and prod. apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease diff --git a/platform/longhorn/storageclasses.yaml b/platform/longhorn/storageclasses.yaml index 5f510b3..925ddac 100644 --- a/platform/longhorn/storageclasses.yaml +++ b/platform/longhorn/storageclasses.yaml @@ -20,7 +20,7 @@ parameters: # (replicaSoftAntiAffinity=false) guarantees these land on distinct nodes, so any single node # loss keeps the volume available. # HARDCODED, not ${LONGHORN_DEFAULT_REPLICA_COUNT}: StorageClass.parameters is map[string]string, - # but kustomize strips the quotes around a bare "${VAR}" placeholder, so Flux postBuild substitution + # but kustomize strips the quotes around a bare "$${VAR}" placeholder, so Flux postBuild substitution # yields an unquoted numeric (3) that server-side apply rejects (int, not string). A quoted numeric # literal like "3"/"30" IS preserved by kustomize. The replica knob still drives the GLOBAL default # via defaultSettings.defaultReplicaCount in helmrelease.yaml (a Helm int value, unaffected). From a344b046c9a06ecb8b7c8b54930b27ad280f2625 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 01:33:16 -0500 Subject: [PATCH 4/7] =?UTF-8?q?docs:=20research=20findings=20=E2=80=94=20m?= =?UTF-8?q?easured=20Flux=20substitution=20semantics=20+=20ranked=20remedi?= =?UTF-8?q?ation=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records three parallel research passes and, importantly, TWO CORRECTIONS. Correction 1 (mine): StrictPostBuildSubstitutions alone is INSUFFICIENT — it fixes nothing about ordering. The actual fix is one line: platform.dependsOn: [flux-system]. Verified against kustomization_controller.go::checkDependencies at our exact version — dependsOn is REVISION-gated, not merely Ready-gated, so a dependent sharing a GitRepository cannot reconcile revision N until the dependency has APPLIED revision N. Correction 2 (a research agent's): the claim that Helm considered the bad release healthy, so remediation never fired, is WRONG — measured on the live cluster, the Helm release secrets show v9 = failed. The machinery did fire. maxUnavailable: 2 of 3 IS confirmed, so the real harm was blast radius, not missed detection. Adds a MEASURED substitution-semantics table, including the two anti-patterns nobody had written down: ${VAR:?msg} renders the MESSAGE as the value, and ${VAR:=default} silently ships the dev value to prod. Both are exempt from strict mode by design — which is exactly why strict would never have caught the gitea/nextcloud dead-guard bug. Also corrects repo doctrine: escaping is TOTAL ($${ARR[0]} is safe); only the unescaped indexed form is fatal. And qualifies an earlier overstatement in this log: kustomize strips structural YAML comments before envsubst, so the hazard is block-scalar bodies, not all comments. Notes Flux 2.4.0 is out of support and that strict became default-on in kustomize-controller v1.9.0 — so the audit is a prerequisite for the upgrade regardless. Ranked plan P0-P5 with what is done, what is ready, and what needs an operator call. --- DECISIONS-LOG.md | 112 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 1e97184..186e673 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1582,3 +1582,115 @@ A Cilium rolling upgrade on this cluster is not risk-free even when the values a `config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is set, and the agent that translates that ClusterIP is the one being replaced. Keep `k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time. + +--- + +## 2026-09-01 — RESEARCH: Flux substitution + HelmRelease best practice. **Measured semantics + ranked plan.** + +Three parallel research passes (substitution/ordering, HelmRelease/CNI, and a full repo audit), +prompted by the incident above. Two of my own earlier conclusions were WRONG and are corrected +here. Everything below marked *measured* was run against flux 2.4.0 / the live cluster; everything +else is cited inference. + +### THE SUBSTITUTION SEMANTICS TABLE (measured — trust this over prose elsewhere) + +| form | behaviour | +|---|---| +| `${DEFINED}` | substituted | +| `${UNDEFINED}` | **silently empty** (non-strict) · fails loudly (strict) | +| `${UNDEFINED:-d}` | **default applied at reconcile time — `--strict` does NOT catch it** | +| `${VAR:?msg}` | renders **the message as the value** — a non-empty WRONG value. ANTI-PATTERN. | +| `${VAR:=192.168.1.50}` | silently ships the dev value to prod. ANTI-PATTERN. | +| `$${VAR}` / `$${VAR:-d}` / `$${ARR[0]}` | literal — **escaping is TOTAL** | +| `$VAR` / `$(cmd)` | untouched, no escaping needed | +| `${...}` / `${a b}` / `${ARR[0]}` unescaped | **hard parse error — fails the whole Kustomization** | + +Flux exempts default-providing operators (`-`, `:-`, `+`, `:+`, `=`, `:=`, `?`, `:?`) from strict +mode by design ("they intentionally handle the unset case"). **That is why strict mode would never +have caught the gitea/nextcloud dead-guard bug**, and why a targeted check was required. + +**Doctrine corrected:** `clusters/base/apps-vms.yaml` claimed an indexed expansion is fatal +"escaped or not". Measured: escaping is total; only the UNESCAPED form is fatal. Fixed in place. + +**Comment nuance:** `kustomize build` strips *structural* YAML comments before envsubst sees them, +so those are usually harmless. But `#` lines INSIDE a block scalar are data and DO reach envsubst, +and a `$`-carrying body comment was observed surviving into built `apps/wave-1`. Do not rely on +stripping — escape in comments too. (An earlier claim in this log that a YAML comment "nearly broke +every merge" was overstated for the structural case; the hazard is real for block-scalar bodies.) + +### CORRECTION 1 (mine) — `StrictPostBuildSubstitutions` alone is INSUFFICIENT + +It fixes nothing about ordering; it converts "render a corrupt value and ship it" into "fail loudly +and keep last-good". **The actual ordering fix is one line**, verified against +`kustomization_controller.go :: checkDependencies` at our exact version (v1.4.0): `dependsOn` is +**revision-gated**, not merely Ready-gated — when the dependency shares a GitRepository, the +dependent cannot reconcile revision N until the dependency has APPLIED revision N. + +`platform` is the only Kustomization in the graph with no `dependsOn`; every other consumer chains +transitively through it. So: + +```yaml +# clusters/base/platform.yaml +spec: + dependsOn: + - name: flux-system # cluster-vars must be applied at THIS revision first +``` + +**Cost, recorded honestly:** the root becomes a gate — a malformed manifest anywhere under +`clusters/dev` now stops the whole graph where today it would not. Fail-closed, consistent with +doctrine, but a real blast-radius change. No deadlock risk: the root runs `wait: false`. + +Rejected alternative: moving `cluster-vars` into the consuming Kustomization is **strictly worse** — +`SubstituteVariables` reads the ConfigMap from the API at build time, *before* the Kustomization +applies its own resources, so it would always read the PREVIOUS value. That converts an occasional +race into a permanent one-reconcile lag. + +### CORRECTION 2 (an agent's) — the safety machinery DID fire + +A research pass concluded that Helm considered the bad release healthy (because +`updateStrategy.rollingUpdate.maxUnavailable: 2` on 3 nodes means `expectedReady = 1`), and +therefore remediation never engaged. **Measured on the live cluster: that is wrong.** The Helm +release secrets show **v9 = `failed`**; Helm's wait DID fail and the release was remediated, which +is why it self-healed. `maxUnavailable: 2 of desired 3` IS confirmed live — so the real harm was +blast radius (2 of 3 agents down at once), not a missed detection. + +### Verified, and worth knowing + +- **Flux 2.4.0 is OUT OF SUPPORT** — the project supports the last three minors; 2.9.x is current. +- **Strict substitution became DEFAULT-ON** in kustomize-controller v1.9.0. So upgrading Flux flips + this behaviour whether or not we opt in — the variable audit is a prerequisite for the upgrade + regardless of today's decision. +- **Cilium's `values.schema.json` permits `k8sServiceHost: ""`** — no `minLength`, no `pattern`. + Helm was never going to reject the bad render. +- **Cilium docs make `k8sServiceHost` REQUIRED under kube-proxy replacement**, precisely because the + environment exports the API ClusterIP and there is no kube-proxy to provision it. +- **The Cilium chart ships zero test templates**, so `spec.test` is useless for it. A post-upgrade + check must live outside Helm. +- `valuesFrom` fails HARD on a missing key (verified in `chartutil/values.go`) unlike postBuild's + silent empty — but only on a MISSING key, not an empty value, and it cannot cross namespaces + (`cluster-vars` is in `flux-system`; the Cilium HelmRelease is in `kube-system`). +- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** It compares the cluster against the + Helm storage manifest — during the incident that manifest WAS the bad render, so drift correction + would have actively re-asserted the broken value. + +### RANKED PLAN + +| P | Action | State | +|---|---|---| +| **P0** | `platform.dependsOn: [flux-system]` — closes the race completely | **PENDING — operator call** (blast-radius change) | +| **P0** | Escape the dead OIDC guards; renderer now refuses the class | ✅ DONE | +| **P1** | Substitution-safety L0 gate, unconditional in `ci/l0.sh` | ✅ DONE — 102 assertions | +| **P1** | Enable `StrictPostBuildSubstitutions=true` | **READY — proven safe**, all 11 paths exit 0 under `flux envsubst --strict`. Sequence AFTER P0. | +| **P2** | `updateStrategy.rollingUpdate.maxUnavailable: 1` on Cilium, as a new **[parity]** field in BOTH the HelmRelease and the Ansible j2 | PENDING — bounds blast radius to one node | +| **P3** | Split Cilium into its own Kustomization (`wait: true`, `interval: 1h`, `timeout: 10m`); `platform` dependsOn it | PENDING — gives `flux suspend kustomization cilium` as an emergency valve | +| **P4** | Upgrade Flux 2.4.0 → 2.9.x | PENDING — out of support; brings strict-by-default | +| **P5** | Cilium MINOR bumps become manual (Renovate PR → human runs the documented pre-flight → merge). Never automerge a CNI minor. | PENDING — process | +| — | Record `${VAR:?}` / `${VAR:=}` as anti-patterns | ✅ DONE (table above) | +| — | *Not recommended*: drift detection on Cilium; `spec.test` on Cilium; `k8sServiceHost: auto` | — | + +### Follow-ups surfaced, not yet actioned +- 6 dead `cluster-vars` knobs; `BASE_DOMAIN` and `CLUSTER_ENV` look like genuine rot. +- `platform/{cilium,longhorn,velero,oauth2-proxy}/values.yaml` reference mirrors carry decorative + `${...}` that can silently drift from the real HelmRelease. +- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time + out while a legitimate Longhorn upgrade is still running. From fdc4a7d9f4a9b15f03184d603d36bd485d1762af Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 01:40:20 -0500 Subject: [PATCH 5/7] =?UTF-8?q?fix(cilium):=20maxUnavailable=202=20->=201?= =?UTF-8?q?=20=E2=80=94=20bound=20CNI=20rollout=20blast=20radius=20(P2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three Cilium agents simultaneously. The agents' `config` init container falls back to the API ClusterIP, which only the Cilium agent itself can translate — and that agent is the one being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`. Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo. maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod that fails to become Ready. The whole value of the setting is that the rollout halts on the first bad node instead of taking the majority of the dataplane with it, and that guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working agent throughout. Set as a [parity] field in ALL THREE places, per this repo's documented contract that the Ansible bootstrap values and the Flux HelmRelease values must agree on settings that define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts it; the two layers fight otherwise): platform/cilium/helmrelease.yaml (authoritative for Flux) infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap) platform/cilium/values.yaml (reference mirror, kept in sync) Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every [parity] field is byte-equal across all three; only k8sServiceHost differs, as designed (${K3S_API_HOST} vs {{ k3s_api_host }}). MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a pod is out of date by hashing spec.template alone, so the pod-template hash does not move, no new ControllerRevision is created, and zero pods are recreated. Helm patches the field in place. (If it were disruptive it would be self-defeating: rolling agents under the old maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with `kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'` plus unchanged pod ages — that is the cleanest proof nothing rolled. Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3 nodes (a minute or two). On a large cluster this becomes painful and a percentage such as "10%" would be right — noted so a future larger cluster revisits it deliberately. SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render still ships; it now breaks one node and stalls rather than taking the majority. platform.dependsOn: [flux-system] (P0) is the fix for the cause. Serves: R-PLAT-01, R-NET-01. Co-Authored-By: Claude Opus 5 (1M context) --- .../cilium/templates/cilium-values.yaml.j2 | 17 +++++++++++++++ platform/cilium/helmrelease.yaml | 21 +++++++++++++++++++ platform/cilium/values.yaml | 10 +++++++++ 3 files changed, 48 insertions(+) diff --git a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 index 0704adb..8a22edf 100644 --- a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 +++ b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 @@ -42,5 +42,22 @@ cgroup: enabled: false hostRoot: /run/cilium/cgroupv2 +# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the +# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a +# 3-node cluster lets one bad render take down two of three agents simultaneously — +# exactly the 2026-09-01 incident, where k8sServiceHost: "" made the agent's `config` +# init container fall back to the in-cluster ClusterIP 10.43.0.1:443. Only the Cilium +# agent can translate that ClusterIP, and the agent in question was the one being +# replaced: `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked. +# A CNI rollout has no healthy-peer fallback — the component being upgraded is the one +# the upgrade needs to reach the API server. This bites hardest on a COLD BOOTSTRAP +# (this role's job), where nothing healthy exists to recover from. With maxUnavailable: 1 +# the DaemonSet will not proceed past an unready pod, bounding damage to a single node. +# Cost: Cilium upgrades serialise and take longer; at 3 nodes that is an acceptable trade. +updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + operator: replicas: {{ cilium_operator_replicas }} diff --git a/platform/cilium/helmrelease.yaml b/platform/cilium/helmrelease.yaml index 781964d..68086d6 100644 --- a/platform/cilium/helmrelease.yaml +++ b/platform/cilium/helmrelease.yaml @@ -89,6 +89,27 @@ spec: autoMount: enabled: false hostRoot: /run/cilium/cgroupv2 + # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident --- + # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1. + # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate. + # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad + # Helm render take down two of three agents simultaneously. That is exactly what happened + # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config` + # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium + # agent itself can translate — and the agent translating it was the one being replaced. + # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once. + # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the + # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet + # will not proceed past an unready pod, so a bad render costs one node, not the cluster. + # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is + # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH). + # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and + # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster + # grows enough that serialised CNI upgrades become an operational problem. + updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field # (Flux/prod may differ). Only the [parity] values above must match the bootstrap. diff --git a/platform/cilium/values.yaml b/platform/cilium/values.yaml index 3223eaf..e9f791b 100644 --- a/platform/cilium/values.yaml +++ b/platform/cilium/values.yaml @@ -42,6 +42,16 @@ cgroup: autoMount: enabled: false hostRoot: /run/cilium/cgroupv2 +# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart +# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the +# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the +# ClusterIP 10.43.0.1:443, translatable only by the very agent being replaced). A CNI upgrade +# has no healthy-peer fallback, least of all on a cold prod bootstrap; maxUnavailable: 1 stops +# the rollout at the first unready pod. Cost: serialised, slower upgrades (fine at 3 nodes). +updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 # NON-parity (Flux/prod may differ; dev sets 1 to save RAM) — omitted from the HelmRelease: # operator.replicas # From 666a60776a981c21d895e1fd079e286135939640 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 01:45:04 -0500 Subject: [PATCH 6/7] =?UTF-8?q?fix(flux):=20close=20the=20substitution=20r?= =?UTF-8?q?ace=20=E2=80=94=20platform=20dependsOn=20root=20+=20strict=20su?= =?UTF-8?q?bstitution=20(P0/P1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/ — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) --- bootstrap/flux/README.md | 60 +++++++++++++++++- bootstrap/flux/bootstrap.sh | 55 +++++++++++++++-- bootstrap/flux/components/kustomization.yaml | 64 ++++++++++++++++++++ clusters/base/platform.yaml | 27 +++++++++ clusters/prod/kustomization.yaml | 10 +++ clusters/prod/sync.yaml | 32 ++++++++++ infra/PROD-PROVISIONING.md | 13 ++++ 7 files changed, 254 insertions(+), 7 deletions(-) create mode 100644 bootstrap/flux/components/kustomization.yaml create mode 100644 clusters/prod/sync.yaml diff --git a/bootstrap/flux/README.md b/bootstrap/flux/README.md index 90465ce..1824ad0 100644 --- a/bootstrap/flux/README.md +++ b/bootstrap/flux/README.md @@ -11,11 +11,69 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux ## Files | File | Purpose | |---|---| -| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. | +| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. | +| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. | | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). | | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). | | `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). | +## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster + +Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git +and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01): + +``` +$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}' +{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system", + "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"} +# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1 +``` + +No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under +`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux +documents for hand-installed clusters ([Customize Flux +manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)): + +``` +flux install --export … -> /gotk-components.yaml # regenerated from the pinned flux CLI +components/kustomization.yaml -> /kustomization.yaml # the only hand-authored part (in git) +kubectl kustomize | kubectl apply -f - +``` + +Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION` +(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations, +more feature gates) is another entry in `components/kustomization.yaml`. + +### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic) + +> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the +> controller Deployment is outside Flux's inventory. Someone must run step 1. + +Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1: + +```bash +# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root +d=$(mktemp -d) +flux install --export --namespace=flux-system \ + --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml" +cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml" +kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict +kubectl -n flux-system rollout status deploy/kustomize-controller +``` + +**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any +reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing +is lost. Confirm afterwards: + +```bash +kubectl -n flux-system get deploy kustomize-controller \ + -o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true +flux get kustomizations -A # all 11 must return to Ready +``` + +**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or +simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off). + ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1) The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker): diff --git a/bootstrap/flux/bootstrap.sh b/bootstrap/flux/bootstrap.sh index 893b19c..33e35c7 100755 --- a/bootstrap/flux/bootstrap.sh +++ b/bootstrap/flux/bootstrap.sh @@ -69,16 +69,59 @@ for bin in flux kubectl; do command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; } done -# --- 1. install Flux controllers --------------------------------------------- +# --- 1. install Flux controllers (rendered through ./components) -------------- # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates, # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly. -if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then - echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})" - flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}" +# +# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently: +# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined +# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see +# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md). +# +# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the +# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux +# documents for hand-installed clusters ("Customize Flux manifests"): +# +# flux install --export > gotk-components.yaml # exact manifests `flux install` would apply +# kustomize overlay = ./components/kustomization.yaml (checked in — the ONLY hand-authored part) +# kubectl apply -k +# +# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION, +# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is +# a temp dir — the repo tree is never written to. +# +# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO +# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so. +echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}" +flux_install_args=(--namespace="${FLUX_NAMESPACE}") +[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}") + +if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then + echo " WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED." + echo " StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN." + flux install "${flux_install_args[@]}" else - echo "==> [1/3] flux install (default controllers only)" - flux install --namespace="${FLUX_NAMESPACE}" + components_dir="${script_dir}/components" + [ -f "${components_dir}/kustomization.yaml" ] || { + echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2 + echo " (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2 + exit 1 + } + render_dir="$(mktemp -d)" + trap 'rm -rf "${render_dir}"' EXIT + echo " rendering flux install --export through ${components_dir}" + flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml" + cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml" + # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux + # version bump) — better a hard stop than silently installing without the feature gate. + kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml" + grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || { + echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2 + echo " Check ./components/kustomization.yaml against this flux CLI version." >&2 + exit 1 + } + kubectl apply -f "${render_dir}/rendered.yaml" fi # --- 2. verify the out-of-band age key Secret exists ------------------------- diff --git a/bootstrap/flux/components/kustomization.yaml b/bootstrap/flux/components/kustomization.yaml new file mode 100644 index 0000000..0a428ec --- /dev/null +++ b/bootstrap/flux/components/kustomization.yaml @@ -0,0 +1,64 @@ +--- +# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments). +# +# WHY THIS DIRECTORY EXISTS +# ========================= +# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the +# reconciliation graph is authored by hand in clusters/base, and the controllers are installed +# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01: +# +# kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}' +# -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1 +# +# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no +# existing mechanism to customize them, and no amount of editing under clusters/** can reach them. +# +# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" — +# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests +# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result. +# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported +# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the +# repo, so the manifests can never drift from the pinned FLUX_VERSION +# (infra/devcontainer/versions.env) the way a vendored copy would. +# +# Every future controller customization (vertical scaling, tolerations, extra feature gates, +# --concurrent) belongs HERE, as another entry under `patches:`. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + # Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in. + - gotk-components.yaml +patches: + # ------------------------------------------------------------------------------------------- + # StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED. + # + # Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the + # EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium + # HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this + # gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state. + # + # PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with + # `kubectl kustomize --load-restrictor LoadRestrictionsNone | flux envsubst --strict` and + # exited 0, with a negative control reproducing the incident. The permanent guard that keeps it + # true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/. + # + # NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) — + # Flux exempts them by design. That class is what the L0 gate covers; the two are complementary. + # + # This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in + # kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in. + # + # JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic + # list, so a strategic-merge patch would REPLACE every arg `flux install` generates + # (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election) + # and silently drop any the CLI adds in a future version. + - target: + group: apps + version: v1 + kind: Deployment + name: kustomize-controller + namespace: flux-system + patch: |- + - op: add + path: /spec/template/spec/containers/0/args/- + value: --feature-gates=StrictPostBuildSubstitutions=true diff --git a/clusters/base/platform.yaml b/clusters/base/platform.yaml index 1f3dec6..02251e1 100644 --- a/clusters/base/platform.yaml +++ b/clusters/base/platform.yaml @@ -13,6 +13,33 @@ spec: timeout: 5m prune: true # drift correction: delete resources removed from git (R-GITOPS-03) wait: true # block `apps` until platform is healthy + # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium). + # + # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root + # `flux-system` Kustomization (clusters//sync.yaml), not by this one. Without an edge here, + # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a + # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING. + # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease. + # + # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against + # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version): + # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile + # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what + # writes a newly-added `cluster-vars` knob — so this one edge closes the race. + # + # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting + # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty, + # velero) chains transitively through it. One edge therefore protects the whole graph. + # + # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest + # ANYWHERE under the root's path (clusters/, which includes clusters/base) fails the root + # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge + # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken + # entrypoint should stop the line, not let consumers render against a half-applied variable set. + # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never + # health-gates its own children (see clusters/dev/sync.yaml). + dependsOn: + - name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars sourceRef: kind: GitRepository name: flux-system diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 5b8b0f9..171eda3 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -5,10 +5,20 @@ # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit. +# +# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD *** +# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution- +# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml` +# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would +# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing +# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml +# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT +# done here: prod is design-only and its root object is the prod bring-up owner's call.) apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base + - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: - target: kind: ConfigMap diff --git a/clusters/prod/sync.yaml b/clusters/prod/sync.yaml new file mode 100644 index 0000000..9130eee --- /dev/null +++ b/clusters/prod/sync.yaml @@ -0,0 +1,32 @@ +--- +# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml). +# +# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars +# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object +# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs +# its own sync.yaml -> ./clusters/prod"; this is it. +# +# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained +# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent +# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no +# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does +# not exist, and NOTHING downstream would ever reconcile. +# +# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches +# Ready on apply success without health-gating its children (platform does its own wait/health +# gating). Do not "harden" this to true. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: flux-system + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + sourceRef: + kind: GitRepository + name: flux-system + path: ./clusters/prod diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index fa959a9..a337754 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -311,6 +311,19 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be own comment flags this as owed "when hardware is built." - [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`. +- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing + the `dependsOn` fix. Only the first is now closed: + - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became + *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries + `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block + forever on a dependency that does not exist and nothing downstream would reconcile. + - [ ] **SOPS decryption** — `clusters/dev/kustomization.yaml` patches every Kustomization with + `decryption: {provider: sops, secretRef: sops-age}`. Prod has no equivalent, so every + SOPS-encrypted Secret would fail to decrypt. Needs the prod age key decision first + (same key as dev, or a prod-specific key — a real security choice, not a copy). + - [ ] **GitRepository source URL** — dev patches in the real Gitea URL. Prod needs its own, + plus a decision on whether prod tracks the same branch as dev or a separate one. + Tracking the same branch means a dev-facing merge reconciles prod. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03). From 6b4da27f841a8193ebb68f95336ea49750a3b336 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 16:27:32 -0500 Subject: [PATCH 7/7] feat(prod): SOPS decryption (same age key as dev) + GitOps source on `main` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator decisions, 2026-09-01: 1. Prod uses the SAME SOPS age key as dev. 2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be decommissioned once prod is deemed stable. SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the overlay references it by name only, never inlines key material. RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod move together. That is a deliberate consequence of dev being temporary, not an oversight, but it removes the "prove it on dev first" safety net that the last few days have leaned on heavily. Worth revisiting if dev outlives its expected life. *** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a cold prod cluster, for two independent reasons found while wiring this: 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for decommission. Prod syncing from it means prod's source of truth lives inside the cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn down. This makes Gitea migration the FIRST task of the decommission plan, not the last. 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository. This is the same split-horizon mechanism that makes dev work, inverted against us. Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and correct; only the host is blocked. Documented at the patch site and in PROD-PROVISIONING.md §7. Gate: MERGE GATE: PASS. Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) --- clusters/prod/kustomization.yaml | 123 +++++++++++++++++++++++++++++++ infra/PROD-PROVISIONING.md | 27 +++++-- 2 files changed, 143 insertions(+), 7 deletions(-) diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 171eda3..ec34c1a 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -20,6 +20,129 @@ resources: - ../base - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: + # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) --- + # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets. + # The `sops-age` Secret is created OUT OF BAND on the prod cluster + # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined. + # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave. + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: platform + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-0 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-1 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-2 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: oauth2-proxy + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: headlamp + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: crafty + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: velero + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) --- + # + # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. *** + # + # Two independent reasons, both consequences of the split-horizon design: + # 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's + # GitOps source lives inside the very cluster slated for decommission. Gitea must move to + # prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. + # 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml + # answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is + # 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve + # gitea.home.keithjackson.dev to itself and never reach a repository. + # + # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster + # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the + # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7. + # + # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev. + # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH + # CLUSTERS. There is no promotion step and no soak window — dev and prod move together. + - patch: |- + apiVersion: source.toolkit.fluxcd.io/v1 + kind: GitRepository + metadata: + name: flux-system + namespace: flux-system + spec: + url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git + ref: + branch: main - target: kind: ConfigMap name: cluster-vars diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index a337754..55202bf 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -317,13 +317,26 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block forever on a dependency that does not exist and nothing downstream would reconcile. - - [ ] **SOPS decryption** — `clusters/dev/kustomization.yaml` patches every Kustomization with - `decryption: {provider: sops, secretRef: sops-age}`. Prod has no equivalent, so every - SOPS-encrypted Secret would fail to decrypt. Needs the prod age key decision first - (same key as dev, or a prod-specific key — a real security choice, not a copy). - - [ ] **GitRepository source URL** — dev patches in the real Gitea URL. Prod needs its own, - plus a decision on whether prod tracks the same branch as dev or a separate one. - Tracking the same branch means a dev-facing merge reconciles prod. + - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key + as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev + exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster + (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only. + - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision + 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and + will be decommissioned once prod is stable. Recorded consequence: once prod is live, + **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window. + The URL is patched in but **cannot work on a cold prod cluster**, for two independent + reasons found 2026-09-01: + 1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for + decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod + loses its source of truth. **This is the decommission plan's first task, not its last.** + 2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire + `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`: + prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve + `gitea.home.keithjackson.dev` to itself and never reach a repository. + Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a + cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own + Phase-6 cutover. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).