During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5