jakman85andClaude Opus 5
fdc4a7d9f4
During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the bootstrap/ entrypoints.
LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.
Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
Prerequisites
tofu applyin../tofu/has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).- Inventory reflects the node IPs.
inventory/hosts.iniships aligned with the LOCKED defaults; regenerate from tofu if overridden:cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini - Run from the dev container (has
kubectl+helm, used by the Cilium/verify/ Flux plays viadelegate_to: localhost).
Run order
cd infra/ansible
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
site.ymlis NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDEsite.ymland must be run separately:ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3(Both take the same target-group override as the rest — see Targeting prod below.)
Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes everything on the host filesystem — including host-level config
that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the
node-host layers or the node comes back missing pieces:
| Re-run | Why it doesn't survive a rebuild |
|---|---|
node-baseline.yml |
packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off |
k3s.yml |
the k3s server binary + config + join (secondaries re-join the VIP) |
kube-vip.yml |
the kube-vip auto-deploy manifest on the server's host path |
vpn.egress.gateway.yml |
the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet |
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip
should return a PIA exit IP (not the home ISP).
What each layer does
| Playbook / role | Action |
|---|---|
k3s.yml / k3s_server |
k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored) |
cilium.yml / cilium |
Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements |
verify.yml / cluster_verify |
waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
flux-bootstrap.yml / flux_bootstrap |
calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev) |
Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config
afterward. The parity-critical values must match between this role's
templates/cilium-values.yaml.j2 and that HelmRelease:
kubeProxyReplacement: truek8sServiceHost: {{ k3s_api_host }}(= primary node IP,10.73.0.11by default)k8sServicePort: 6443
The HelmRelease currently has k8sServiceHost: "REPLACE-ME" — WS-NET must set it
to the same value this role uses (surfaced by tofu output primary_node_ip).
Interface assumed of bootstrap/ (authored by other workers, NOT here)
roles/flux_bootstrap invokes two entrypoints via overridable command vars:
flux_bootstrap_sops_cmd→bootstrap/sops/install-age-key.sh(Security Engineer, T-1.4): installs the age private key as thesops-ageSecret influx-system. Key never committed (.gitignoreguardsage.key/*.key).flux_bootstrap_flux_cmd→bootstrap/flux/bootstrap.sh(GitOps Engineer, T-1.4): runsflux bootstrap ... --path=clusters/dev.
These script names/args are assumptions — if the bootstrap deliverables
expose different entrypoints, override the two vars in
roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged
for PM reconciliation.
Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
inventory/hosts.ini carries a [k3s_prod_primary] / [k3s_prod_secondary] /
[k3s_prod:children] section for the three planned bare-metal nodes (k3s-prod-1 at
192.168.2.10 exists and is reachable today; k3s-prod-2/-3 at .11/.12 are addressed but
not yet racked, so their inventory lines stay commented out), with its own group_vars/k3s_prod/
(vaulted ansible_become_password + non-secret overrides).
Nothing here is forked for prod. Every node-host play takes its target group from a variable that defaults to the dev group, so the dev commands above are unchanged, and prod is the same command plus an override. Two separate mechanisms, for two separate reasons:
| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play runs on | k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts, vpn_egress_gateway_hosts |
-e extra vars only |
Ansible templates a play's hosts: before it has any host, so it cannot see group_vars at all |
| Which group a role means by "my primary" / "my servers" | k3s_primary_group, k3s_servers_group (+ cluster_kubeconfig_name) |
group_vars — dev in all.yml, prod in k3s_prod/vars.yml |
Host context exists by then, so each environment answers for itself with no per-run flag — same mechanism k3s_api_host already used |
playbooks/node-health.yml established the first pattern (-e node_health_hosts=k3s_prod); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml
# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
-e k3s_hosts=k3s_prod \
-e k3s_primary_hosts=k3s_prod_primary \
-e k3s_secondary_hosts=k3s_prod_secondary
What the roles then resolve by themselves for a prod host, from group_vars/k3s_prod/vars.yml:
k3s_primary_group=k3s_prod_primary (the --cluster-init founder, the node-token delegate_to
target, and the node Cilium fetches the kubeconfig from), k3s_servers_group=k3s_prod (the tls-san
node-IP list), k3s_api_host=192.168.2.10, kube_vip_address=192.168.2.9, and
cluster_kubeconfig_name=prod.kubeconfig — so a prod run can never overwrite the
.cache/dev.kubeconfig that infra/devcontainer/dev.sh exports as $KUBECONFIG.
Packages-only baseline (--tags packages)
roles/node_baseline is tagged packages / system / disks. An untagged run is unchanged (all
three, in order). --tags packages runs only the apt task:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
This exists to break a circular dependency: prod's disk layout (infra/PROD-PROVISIONING.md §2) is
blocked on a SMART read, SMART needs smartmontools, smartmontools ships in
node_baseline_packages — but the role's disk tasks need block devices that decision hasn't chosen
yet. With --tags packages no disks-tagged task is selected, so the deliberate
/dev/BLOCKED-pending-disk-decision-… placeholders in group_vars/k3s_prod/vars.yml are never
even templated, let alone handed to mkfs. A full run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
Still not runnable end to end
- The disk-layout (§2) and the
node_baseline_*_deviceoverrides remain BLOCKED placeholders. group_vars/k3s_prod/vault.yml(the become password) does not exist yet, so no prod play can escalate — and--checkagainst prod fails for the same reason.flux-bootstrap.ymlretargets, butroles/flux_bootstrapshells out tobootstrap/flux/bootstrap.sh, which bootstraps--path=clusters/dev. Prod needs--path=clusters/prod: abootstrap/change, not a target-group change (§7).vpn.egress.gateway.ymlis parameterised (-e vpn_egress_gateway_hosts=…) but prod has no egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own vaulted PIA credentials.- The operator has ruled the prod cluster stays uninitialized until all three nodes are present
(§8).
192.168.2.20is prod's ingress Gateway VIP — a Flux/cluster-varsconcern, not an Ansible one; the kube-vip control-plane VIP is192.168.2.9(kube_vip_address).
Everything above this section describes the dev runbook; the values in it (192.168.1.x,
.cache/dev.kubeconfig, clusters/dev) are dev's, supplied by group_vars, not baked into the
roles.
Static validation (PM, in dev container — NOT run here)
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/