jakman85andClaude Opus 5
6f61c9622d
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
217 lines
14 KiB
Markdown
217 lines
14 KiB
Markdown
# infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
|
|
|
|
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
|
|
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
|
|
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.
|
|
|
|
> **LIVE.** These plays have been executed against the running dev cluster (k3s
|
|
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
|
|
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.
|
|
|
|
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
|
|
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
|
|
|
|
## Prerequisites
|
|
1. `tofu apply` in `../tofu/` has created the VMs; cloud-init (`../cloud-init/`)
|
|
has baselined them (modules, sysctls, Longhorn disk, SSH user).
|
|
2. Inventory reflects the node IPs. `inventory/hosts.ini` ships aligned with the
|
|
LOCKED defaults; regenerate from tofu if overridden:
|
|
`cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
|
|
3. Run from the dev container (has `kubectl` + `helm`, used by the Cilium/verify/
|
|
Flux plays via `delegate_to: localhost`).
|
|
|
|
## Run order
|
|
```
|
|
cd infra/ansible
|
|
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
|
|
# — or step by step —
|
|
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
|
|
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
|
|
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
|
|
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
|
|
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
|
|
```
|
|
|
|
> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
|
|
> layers live OUTSIDE `site.yml` and must be run separately:
|
|
> ```
|
|
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers
|
|
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
|
|
> ```
|
|
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
|
|
|
|
> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
|
|
> ```
|
|
> ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health
|
|
> ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
|
|
> ```
|
|
> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
|
|
> until `-e memattr_survey_reboots_authorised=true` is passed. See
|
|
> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
|
|
|
|
## Re-provisioning a node (tofu -replace / fresh OS)
|
|
A node rebuild wipes **everything on the host filesystem** — including host-level config
|
|
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
|
|
node-host layers or the node comes back missing pieces:
|
|
|
|
| Re-run | Why it doesn't survive a rebuild |
|
|
|---|---|
|
|
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
|
|
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
|
|
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
|
|
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |
|
|
|
|
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
|
|
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
|
|
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
|
|
should return a PIA exit IP (not the home ISP).
|
|
|
|
## What each layer does
|
|
| Playbook / role | Action |
|
|
|---|---|
|
|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
|
|
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
|
|
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
|
|
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |
|
|
|
|
## Cilium ownership / parity contract (docs/02)
|
|
Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
|
|
afterward. The parity-critical values **must match** between this role's
|
|
`templates/cilium-values.yaml.j2` and that HelmRelease:
|
|
- `kubeProxyReplacement: true`
|
|
- `k8sServiceHost: {{ k3s_apiserver_endpoint }}` — **the kube-vip CONTROL-PLANE VIP**
|
|
(`= kube_vip_address`: dev `192.168.1.49`, prod `192.168.2.9`), **not** `k3s_api_host`
|
|
- `k8sServicePort: {{ k3s_api_port }}` (6443 in both environments)
|
|
|
|
### Two API addresses, split 2026-09-06 — do not re-merge them
|
|
`k3s_api_host` and `k3s_apiserver_endpoint` were one variable, and the conflation put every
|
|
prod `cilium-agent` on `192.168.2.10` (`k3s-prod-1`'s own address) on a cluster whose point is
|
|
HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
|
|
service or policy update ever lands, which is a silent failure, not an outage.
|
|
|
|
| variable | value | consumed by | why it must be that address |
|
|
|---|---|---|---|
|
|
| `k3s_api_host` | the founder's own node IP (dev `192.168.1.50`, prod `192.168.2.10`) | `roles/k3s_server`: the secondary `--server https://…:6443` join URL, and one `k3s_tls_sans` entry | on a **cold** build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran `--cluster-init` and is up |
|
|
| `k3s_apiserver_endpoint` | `{{ kube_vip_address }}` (dev `192.168.1.49`, prod `192.168.2.9`) | `roles/cilium` → `k8sServiceHost`; any steady-state client | it must survive losing any single server |
|
|
|
|
Consequences already handled: `playbooks/site.yml` imports `kube-vip.yml` **between** `k3s.yml`
|
|
and `cilium.yml` (the VIP has to be floating before the CNI dials it), and `roles/kube_vip`'s
|
|
last task blocks until the VIP answers on 6443. `k3s_tls_sans` already carried both addresses,
|
|
so no cert reissue and no k3s restart are involved.
|
|
|
|
The Flux side of the same contract is `${K3S_API_HOST}` in `platform/cilium/helmrelease.yaml`,
|
|
set per overlay in `clusters/<env>/kustomization.yaml`; `[parity]` means it must equal
|
|
`k3s_apiserver_endpoint` for the SAME environment. Asserted by
|
|
`conformance/tests/l0-static/ha-control-plane-as-code`.
|
|
|
|
## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
|
|
`roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
|
|
- `flux_bootstrap_sops_cmd` → `bootstrap/sops/install-age-key.sh` (Security
|
|
Engineer, T-1.4): installs the age **private** key as the `sops-age` Secret in
|
|
`flux-system`. Key never committed (`.gitignore` guards `age.key`/`*.key`).
|
|
- `flux_bootstrap_flux_cmd` → `bootstrap/flux/bootstrap.sh` (GitOps Engineer,
|
|
T-1.4): runs `flux bootstrap ... --path=clusters/dev`.
|
|
|
|
**These script names/args are assumptions** — if the bootstrap deliverables
|
|
expose different entrypoints, override the two vars in
|
|
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
|
|
for PM reconciliation.
|
|
|
|
## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
|
|
|
|
`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
|
|
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
|
|
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
|
|
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
|
|
(vaulted `ansible_become_password` + non-secret overrides).
|
|
|
|
**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
|
|
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
|
|
command plus an override. Two separate mechanisms, for two separate reasons:
|
|
|
|
| Layer | Knob | Set via | Why |
|
|
|---|---|---|---|
|
|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
|
|
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |
|
|
|
|
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
|
|
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
|
|
bring-up chain.
|
|
`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
|
|
for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
|
|
role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
|
|
can be mistyped; inventory membership cannot.
|
|
|
|
```bash
|
|
# dev — completely unchanged, no extra vars
|
|
ansible-playbook playbooks/site.yml
|
|
|
|
# prod — one flag for the whole-cluster plays
|
|
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
|
|
|
|
# prod — k3s.yml needs both halves of the primary/secondary split
|
|
ansible-playbook playbooks/k3s.yml \
|
|
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
|
|
|
|
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
|
|
ansible-playbook playbooks/site.yml \
|
|
-e k3s_hosts=k3s_prod \
|
|
-e k3s_primary_hosts=k3s_prod_primary \
|
|
-e k3s_secondary_hosts=k3s_prod_secondary
|
|
```
|
|
|
|
What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
|
|
`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
|
|
target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
|
|
node-IP list), `k3s_api_host=192.168.2.10` (the founder, bootstrap-join only),
|
|
`kube_vip_address=192.168.2.9` — and therefore `k3s_apiserver_endpoint=192.168.2.9`, which is what
|
|
Cilium dials — and
|
|
`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
|
|
`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
|
|
|
|
### Packages-only baseline (`--tags packages`)
|
|
`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
|
|
three, in order). `--tags packages` runs **only** the apt task:
|
|
|
|
```bash
|
|
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
|
|
```
|
|
|
|
This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
|
|
blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
|
|
`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
|
|
yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
|
|
`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
|
|
even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
|
|
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
|
|
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
|
|
|
|
### Still not runnable end to end
|
|
- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
|
|
- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
|
|
escalate — and `--check` against prod fails for the same reason.
|
|
- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
|
|
`bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
|
|
`--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
|
|
- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
|
|
egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
|
|
vaulted PIA credentials.
|
|
- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
|
|
(§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
|
|
Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).
|
|
|
|
Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
|
|
`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
|
|
roles.
|
|
|
|
## Static validation (PM, in dev container — NOT run here)
|
|
```
|
|
yamllint -c ../../ci/yamllint.yaml infra/ansible
|
|
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
|
|
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/
|
|
```
|