diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 63b347c..1e97184 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1496,3 +1496,89 @@ thing that ever closes a requirement. chart default rather than set explicitly — see the follow-up in `platform/longhorn/`. - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI. + +--- + +## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.** + +Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put +`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes** +and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down). +Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold +prod bootstrap than it was here. + +### What happened + +The HelmRelease config digest tells the whole story: + +``` +v8 05:49:39 digest e83bf695… <- before +v9 05:50:11 digest eebd6c88… <- the bad render +v10 05:55:23 digest e83bf695… <- identical to v8 again +``` + +The config changed and then changed **back to byte-identical**. Nothing in git differed +semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus +`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets +`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady +state is identical. The damage happened in a transient intermediate state. + +### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations + +- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization. +- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`** + Kustomization, via `postBuild.substituteFrom: cluster-vars`. +- There is **no dependency ordering between defining a variable and using it.** + +So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`, +which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an +undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm +upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` — +which only the Cilium agent itself can translate. The agent was the thing restarting, so its own +`config` init container could not reach the API server: + +``` +level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443 + error="dial tcp 10.43.0.1:443: i/o timeout" +level=fatal msg="Build config failed" +``` + +Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the +correct value, digest returned to `e83bf695…`, and all three agents came up. + +`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires +the ConfigMap to *exist*, not that every referenced key is defined. + +### Why this matters much more for prod + +On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the +race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**, +there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may +never come up at all — with no healthy peer to recover from. This is a plausible +bring-up-blocker for `clusters/prod`. + +### The fix — make substitution FAIL CLOSED + +`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict +substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail +the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine +("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts +this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile +failure*. + +**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization +that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in +embedded scripts. The safe sequence, before flipping it: +1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in + `cluster-vars` or correctly escaped as `$${VAR}`. +2. Then enable the gate, and let CI + a dev reconcile prove it. + +Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and +reference it in a later one**, so the definition is always reconciled before its first use. + +### Operational note worth keeping + +A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the +`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is +set, and the agent that translates that ClusterIP is the one being replaced. Keep +`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time.