jakman85andClaude Opus 5
cf33338448
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the bootstrap/ entrypoints.
LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.
Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
Prerequisites
tofu applyin../tofu/has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).- Inventory reflects the node IPs.
inventory/hosts.iniships aligned with the LOCKED defaults; regenerate from tofu if overridden:cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini - Run from the dev container (has
kubectl+helm, used by the Cilium/verify/ Flux plays viadelegate_to: localhost).
Run order
cd infra/ansible
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
site.ymlis NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDEsite.ymland must be run separately:ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3(Both take the same target-group override as the rest — see Targeting prod below.)
Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes everything on the host filesystem — including host-level config
that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the
node-host layers or the node comes back missing pieces:
| Re-run | Why it doesn't survive a rebuild |
|---|---|
node-baseline.yml |
packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off |
k3s.yml |
the k3s server binary + config + join (secondaries re-join the VIP) |
kube-vip.yml |
the kube-vip auto-deploy manifest on the server's host path |
vpn.egress.gateway.yml |
the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet |
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip
should return a PIA exit IP (not the home ISP).
What each layer does
| Playbook / role | Action |
|---|---|
k3s.yml / k3s_server |
k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored) |
cilium.yml / cilium |
Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements |
verify.yml / cluster_verify |
waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
flux-bootstrap.yml / flux_bootstrap |
calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev) |
Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config
afterward. The parity-critical values must match between this role's
templates/cilium-values.yaml.j2 and that HelmRelease:
kubeProxyReplacement: truek8sServiceHost: {{ k3s_api_host }}(= primary node IP,10.73.0.11by default)k8sServicePort: 6443
The HelmRelease currently has k8sServiceHost: "REPLACE-ME" — WS-NET must set it
to the same value this role uses (surfaced by tofu output primary_node_ip).
Interface assumed of bootstrap/ (authored by other workers, NOT here)
roles/flux_bootstrap invokes two entrypoints via overridable command vars:
flux_bootstrap_sops_cmd→bootstrap/sops/install-age-key.sh(Security Engineer, T-1.4): installs the age private key as thesops-ageSecret influx-system. Key never committed (.gitignoreguardsage.key/*.key).flux_bootstrap_flux_cmd→bootstrap/flux/bootstrap.sh(GitOps Engineer, T-1.4): runsflux bootstrap ... --path=clusters/dev.
These script names/args are assumptions — if the bootstrap deliverables
expose different entrypoints, override the two vars in
roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged
for PM reconciliation.
Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
inventory/hosts.ini carries a [k3s_prod_primary] / [k3s_prod_secondary] /
[k3s_prod:children] section for the three planned bare-metal nodes (k3s-prod-1 at
192.168.2.10 exists and is reachable today; k3s-prod-2/-3 at .11/.12 are addressed but
not yet racked, so their inventory lines stay commented out), with its own group_vars/k3s_prod/
(vaulted ansible_become_password + non-secret overrides).
Nothing here is forked for prod. Every node-host play takes its target group from a variable that defaults to the dev group, so the dev commands above are unchanged, and prod is the same command plus an override. Two separate mechanisms, for two separate reasons:
| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play runs on | k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts, vpn_egress_gateway_hosts |
-e extra vars only |
Ansible templates a play's hosts: before it has any host, so it cannot see group_vars at all |
| Which group a role means by "my primary" / "my servers" | k3s_primary_group, k3s_servers_group (+ cluster_kubeconfig_name) |
group_vars — dev in all.yml, prod in k3s_prod/vars.yml |
Host context exists by then, so each environment answers for itself with no per-run flag — same mechanism k3s_api_host already used |
playbooks/node-health.yml established the first pattern (-e node_health_hosts=k3s_prod); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml
# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
-e k3s_hosts=k3s_prod \
-e k3s_primary_hosts=k3s_prod_primary \
-e k3s_secondary_hosts=k3s_prod_secondary
What the roles then resolve by themselves for a prod host, from group_vars/k3s_prod/vars.yml:
k3s_primary_group=k3s_prod_primary (the --cluster-init founder, the node-token delegate_to
target, and the node Cilium fetches the kubeconfig from), k3s_servers_group=k3s_prod (the tls-san
node-IP list), k3s_api_host=192.168.2.10, kube_vip_address=192.168.2.9, and
cluster_kubeconfig_name=prod.kubeconfig — so a prod run can never overwrite the
.cache/dev.kubeconfig that infra/devcontainer/dev.sh exports as $KUBECONFIG.
Packages-only baseline (--tags packages)
roles/node_baseline is tagged packages / system / disks. An untagged run is unchanged (all
three, in order). --tags packages runs only the apt task:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
This exists to break a circular dependency: prod's disk layout (infra/PROD-PROVISIONING.md §2) is
blocked on a SMART read, SMART needs smartmontools, smartmontools ships in
node_baseline_packages — but the role's disk tasks need block devices that decision hasn't chosen
yet. With --tags packages no disks-tagged task is selected, so the deliberate
/dev/BLOCKED-pending-disk-decision-… placeholders in group_vars/k3s_prod/vars.yml are never
even templated, let alone handed to mkfs. A full run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
Still not runnable end to end
- The disk-layout (§2) and the
node_baseline_*_deviceoverrides remain BLOCKED placeholders. group_vars/k3s_prod/vault.yml(the become password) does not exist yet, so no prod play can escalate — and--checkagainst prod fails for the same reason.flux-bootstrap.ymlretargets, butroles/flux_bootstrapshells out tobootstrap/flux/bootstrap.sh, which bootstraps--path=clusters/dev. Prod needs--path=clusters/prod: abootstrap/change, not a target-group change (§7).vpn.egress.gateway.ymlis parameterised (-e vpn_egress_gateway_hosts=…) but prod has no egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own vaulted PIA credentials.- The operator has ruled the prod cluster stays uninitialized until all three nodes are present
(§8).
192.168.2.20is prod's ingress Gateway VIP — a Flux/cluster-varsconcern, not an Ansible one; the kube-vip control-plane VIP is192.168.2.9(kube_vip_address).
Everything above this section describes the dev runbook; the values in it (192.168.1.x,
.cache/dev.kubeconfig, clusters/dev) are dev's, supplied by group_vars, not baked into the
roles.
Static validation (PM, in dev container — NOT run here)
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/