Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
Files
4f37fc375ee783776febd02b58b19ab2fc0a5035
home-network-k3s-impl/infra/ansible/README.md
T
jakman85andClaude Opus 5 8e913fdd7b fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results 
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.

REFINED MECHANISM — two kernel functions were conflated:
  efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
    (signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
    reserves a modest wrong amount and costs nothing noticeable.
  efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
    'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.

THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.

STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.

SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.

SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.

Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.

Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-03 23:50:59 -05:00

12 KiB
Raw Blame History

infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)

The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off (T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the CNI, verifies readiness, then invokes the bootstrap/ entrypoints.

LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.

Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).

Prerequisites

  1. tofu apply in ../tofu/ has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).
  2. Inventory reflects the node IPs. inventory/hosts.ini ships aligned with the LOCKED defaults; regenerate from tofu if overridden: cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini
  3. Run from the dev container (has kubectl + helm, used by the Cilium/verify/ Flux plays via delegate_to: localhost).

Run order

cd infra/ansible
ansible-playbook playbooks/site.yml        # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml    # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml         # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml      # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml      # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOPS then Flux)

site.yml is NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDE site.yml and must be run separately:

ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); defaults to k3s_servers
ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3

(Both take the same target-group override as the rest — see Targeting prod below.)

Standalone diagnostics (not part of any bring-up; safe to point anywhere):

ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey

memattr-survey.yml is the only play here that reboots its targets, and it will not do so until -e memattr_survey_reboots_authorised=true is passed. See roles/memattr_survey/README.md and infra/PROD-PROVISIONING.md §0b.

Re-provisioning a node (tofu -replace / fresh OS)

A node rebuild wipes everything on the host filesystem — including host-level config that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the node-host layers or the node comes back missing pieces:

Re-run Why it doesn't survive a rebuild
node-baseline.yml packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off
k3s.yml the k3s server binary + config + join (secondaries re-join the VIP)
kube-vip.yml the kube-vip auto-deploy manifest on the server's host path
vpn.egress.gateway.yml the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet

Everything Flux owns (platform + apps) self-heals once the node re-joins; only the node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3, systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip should return a PIA exit IP (not the home ISP).

What each layer does

Playbook / role Action
k3s.yml / k3s_server k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored)
cilium.yml / cilium Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements
verify.yml / cluster_verify waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent
flux-bootstrap.yml / flux_bootstrap calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev)

Cilium ownership / parity contract (docs/02)

Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config afterward. The parity-critical values must match between this role's templates/cilium-values.yaml.j2 and that HelmRelease:

  • kubeProxyReplacement: true
  • k8sServiceHost: {{ k3s_api_host }} (= primary node IP, 10.73.0.11 by default)
  • k8sServicePort: 6443

The HelmRelease currently has k8sServiceHost: "REPLACE-ME" — WS-NET must set it to the same value this role uses (surfaced by tofu output primary_node_ip).

Interface assumed of bootstrap/ (authored by other workers, NOT here)

roles/flux_bootstrap invokes two entrypoints via overridable command vars:

  • flux_bootstrap_sops_cmd → bootstrap/sops/install-age-key.sh (Security Engineer, T-1.4): installs the age private key as the sops-age Secret in flux-system. Key never committed (.gitignore guards age.key/*.key).
  • flux_bootstrap_flux_cmd → bootstrap/flux/bootstrap.sh (GitOps Engineer, T-1.4): runs flux bootstrap ... --path=clusters/dev.

These script names/args are assumptions — if the bootstrap deliverables expose different entrypoints, override the two vars in roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged for PM reconciliation.

Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)

inventory/hosts.ini carries a [k3s_prod_primary] / [k3s_prod_secondary] / [k3s_prod:children] section for the three planned bare-metal nodes (k3s-prod-1 at 192.168.2.10 exists and is reachable today; k3s-prod-2/-3 at .11/.12 are addressed but not yet racked, so their inventory lines stay commented out), with its own group_vars/k3s_prod/ (vaulted ansible_become_password + non-secret overrides).

Nothing here is forked for prod. Every node-host play takes its target group from a variable that defaults to the dev group, so the dev commands above are unchanged, and prod is the same command plus an override. Two separate mechanisms, for two separate reasons:

Layer Knob Set via Why
Which hosts a play runs on k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts, vpn_egress_gateway_hosts -e extra vars only Ansible templates a play's hosts: before it has any host, so it cannot see group_vars at all
Which group a role means by "my primary" / "my servers" k3s_primary_group, k3s_servers_group (+ cluster_kubeconfig_name) group_vars — dev in all.yml, prod in k3s_prod/vars.yml Host context exists by then, so each environment answers for itself with no per-run flag — same mechanism k3s_api_host already used

playbooks/node-health.yml established the first pattern (-e node_health_hosts=k3s_prod); it keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the bring-up chain. playbooks/memattr-survey.yml follows it (-e memattr_survey_hosts=…, defaulting to k3s_prod) for the same reason, and adds a second interlock on top: because it reboots its targets, the role also refuses any host that is not a member of memattr_survey_required_group. A host pattern can be mistyped; inventory membership cannot.

# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml

# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml   -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml        -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml  -e k3s_hosts=k3s_prod   # see caveat below

# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
  -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary

# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
  -e k3s_hosts=k3s_prod \
  -e k3s_primary_hosts=k3s_prod_primary \
  -e k3s_secondary_hosts=k3s_prod_secondary

What the roles then resolve by themselves for a prod host, from group_vars/k3s_prod/vars.yml: k3s_primary_group=k3s_prod_primary (the --cluster-init founder, the node-token delegate_to target, and the node Cilium fetches the kubeconfig from), k3s_servers_group=k3s_prod (the tls-san node-IP list), k3s_api_host=192.168.2.10, kube_vip_address=192.168.2.9, and cluster_kubeconfig_name=prod.kubeconfig — so a prod run can never overwrite the .cache/dev.kubeconfig that infra/devcontainer/dev.sh exports as $KUBECONFIG.

Packages-only baseline (--tags packages)

roles/node_baseline is tagged packages / system / disks. An untagged run is unchanged (all three, in order). --tags packages runs only the apt task:

ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1

This exists to break a circular dependency: prod's disk layout (infra/PROD-PROVISIONING.md §2) is blocked on a SMART read, SMART needs smartmontools, smartmontools ships in node_baseline_packages — but the role's disk tasks need block devices that decision hasn't chosen yet. With --tags packages no disks-tagged task is selected, so the deliberate /dev/BLOCKED-pending-disk-decision-… placeholders in group_vars/k3s_prod/vars.yml are never even templated, let alone handed to mkfs. A full run against a node whose devices are still placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).

Still not runnable end to end

  • The disk-layout (§2) and the node_baseline_*_device overrides remain BLOCKED placeholders.
  • group_vars/k3s_prod/vault.yml (the become password) does not exist yet, so no prod play can escalate — and --check against prod fails for the same reason.
  • flux-bootstrap.yml retargets, but roles/flux_bootstrap shells out to bootstrap/flux/bootstrap.sh, which bootstraps --path=clusters/dev. Prod needs --path=clusters/prod: a bootstrap/ change, not a target-group change (§7).
  • vpn.egress.gateway.yml is parameterised (-e vpn_egress_gateway_hosts=…) but prod has no egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own vaulted PIA credentials.
  • The operator has ruled the prod cluster stays uninitialized until all three nodes are present (§8). 192.168.2.20 is prod's ingress Gateway VIP — a Flux/cluster-vars concern, not an Ansible one; the kube-vip control-plane VIP is 192.168.2.9 (kube_vip_address).

Everything above this section describes the dev runbook; the values in it (192.168.1.x, .cache/dev.kubeconfig, clusters/dev) are dev's, supplied by group_vars, not baked into the roles.

Static validation (PM, in dev container — NOT run here)

yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/          # if the profile is enforced for infra/
Reference in new issue View Git Blame Copy Permalink