jakman85andClaude Opus 5
8e913fdd7b
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
196 lines
12 KiB
Markdown
196 lines
12 KiB
Markdown
# infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
|
|
|
|
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
|
|
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
|
|
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.
|
|
|
|
> **LIVE.** These plays have been executed against the running dev cluster (k3s
|
|
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
|
|
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.
|
|
|
|
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
|
|
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
|
|
|
|
## Prerequisites
|
|
1. `tofu apply` in `../tofu/` has created the VMs; cloud-init (`../cloud-init/`)
|
|
has baselined them (modules, sysctls, Longhorn disk, SSH user).
|
|
2. Inventory reflects the node IPs. `inventory/hosts.ini` ships aligned with the
|
|
LOCKED defaults; regenerate from tofu if overridden:
|
|
`cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
|
|
3. Run from the dev container (has `kubectl` + `helm`, used by the Cilium/verify/
|
|
Flux plays via `delegate_to: localhost`).
|
|
|
|
## Run order
|
|
```
|
|
cd infra/ansible
|
|
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
|
|
# — or step by step —
|
|
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
|
|
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
|
|
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
|
|
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
|
|
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
|
|
```
|
|
|
|
> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
|
|
> layers live OUTSIDE `site.yml` and must be run separately:
|
|
> ```
|
|
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers
|
|
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
|
|
> ```
|
|
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
|
|
|
|
> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
|
|
> ```
|
|
> ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health
|
|
> ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
|
|
> ```
|
|
> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
|
|
> until `-e memattr_survey_reboots_authorised=true` is passed. See
|
|
> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
|
|
|
|
## Re-provisioning a node (tofu -replace / fresh OS)
|
|
A node rebuild wipes **everything on the host filesystem** — including host-level config
|
|
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
|
|
node-host layers or the node comes back missing pieces:
|
|
|
|
| Re-run | Why it doesn't survive a rebuild |
|
|
|---|---|
|
|
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
|
|
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
|
|
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
|
|
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |
|
|
|
|
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
|
|
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
|
|
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
|
|
should return a PIA exit IP (not the home ISP).
|
|
|
|
## What each layer does
|
|
| Playbook / role | Action |
|
|
|---|---|
|
|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
|
|
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
|
|
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
|
|
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |
|
|
|
|
## Cilium ownership / parity contract (docs/02)
|
|
Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
|
|
afterward. The parity-critical values **must match** between this role's
|
|
`templates/cilium-values.yaml.j2` and that HelmRelease:
|
|
- `kubeProxyReplacement: true`
|
|
- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
|
|
- `k8sServicePort: 6443`
|
|
|
|
The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
|
|
to the same value this role uses (surfaced by `tofu output primary_node_ip`).
|
|
|
|
## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
|
|
`roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
|
|
- `flux_bootstrap_sops_cmd` → `bootstrap/sops/install-age-key.sh` (Security
|
|
Engineer, T-1.4): installs the age **private** key as the `sops-age` Secret in
|
|
`flux-system`. Key never committed (`.gitignore` guards `age.key`/`*.key`).
|
|
- `flux_bootstrap_flux_cmd` → `bootstrap/flux/bootstrap.sh` (GitOps Engineer,
|
|
T-1.4): runs `flux bootstrap ... --path=clusters/dev`.
|
|
|
|
**These script names/args are assumptions** — if the bootstrap deliverables
|
|
expose different entrypoints, override the two vars in
|
|
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
|
|
for PM reconciliation.
|
|
|
|
## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
|
|
|
|
`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
|
|
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
|
|
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
|
|
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
|
|
(vaulted `ansible_become_password` + non-secret overrides).
|
|
|
|
**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
|
|
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
|
|
command plus an override. Two separate mechanisms, for two separate reasons:
|
|
|
|
| Layer | Knob | Set via | Why |
|
|
|---|---|---|---|
|
|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
|
|
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |
|
|
|
|
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
|
|
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
|
|
bring-up chain.
|
|
`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
|
|
for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
|
|
role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
|
|
can be mistyped; inventory membership cannot.
|
|
|
|
```bash
|
|
# dev — completely unchanged, no extra vars
|
|
ansible-playbook playbooks/site.yml
|
|
|
|
# prod — one flag for the whole-cluster plays
|
|
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
|
|
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
|
|
|
|
# prod — k3s.yml needs both halves of the primary/secondary split
|
|
ansible-playbook playbooks/k3s.yml \
|
|
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
|
|
|
|
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
|
|
ansible-playbook playbooks/site.yml \
|
|
-e k3s_hosts=k3s_prod \
|
|
-e k3s_primary_hosts=k3s_prod_primary \
|
|
-e k3s_secondary_hosts=k3s_prod_secondary
|
|
```
|
|
|
|
What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
|
|
`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
|
|
target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
|
|
node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
|
|
`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
|
|
`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
|
|
|
|
### Packages-only baseline (`--tags packages`)
|
|
`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
|
|
three, in order). `--tags packages` runs **only** the apt task:
|
|
|
|
```bash
|
|
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
|
|
```
|
|
|
|
This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
|
|
blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
|
|
`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
|
|
yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
|
|
`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
|
|
even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
|
|
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
|
|
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
|
|
|
|
### Still not runnable end to end
|
|
- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
|
|
- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
|
|
escalate — and `--check` against prod fails for the same reason.
|
|
- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
|
|
`bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
|
|
`--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
|
|
- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
|
|
egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
|
|
vaulted PIA credentials.
|
|
- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
|
|
(§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
|
|
Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).
|
|
|
|
Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
|
|
`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
|
|
roles.
|
|
|
|
## Static validation (PM, in dev container — NOT run here)
|
|
```
|
|
yamllint -c ../../ci/yamllint.yaml infra/ansible
|
|
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
|
|
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/
|
|
```
|