jakman85andClaude Opus 5
a984442cae
Two debt items, both about making a latent problem impossible rather than fixed. === 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) === The prod nodes disagreed, measured from k3s's own etcd snapshot annotations: prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless — Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes apart, which is correct staggering — but it makes cross-node log correlation actively misleading, and this cluster has had two unplanned reboots in two days. `Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records "no other service in the repo sets a timezone ... so use the safe default" with TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact: THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than inventing a convention. UTC also removes the class — no DST discontinuity, node time == platform time. New knob `node_baseline_timezone`, overridable per environment via group_vars (the same seam as node_baseline_backports_kernel) but deliberately NOT restated there: no environment needs to differ, and a second copy is a second thing to disagree. Guarded by an assert in the role's existing BLOCKED style before the module runs. `--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the offset evidence, and is the whole verification — the play was never run for real: k3s-prod-1: -"America/Chicago" +"Etc/UTC" changed k3s-prod-2: -"America/Chicago" +"Etc/UTC" changed k3s-prod-3: -"America/New_York" +"Etc/UTC" changed The same --check against DEV reports the task `ok` on all three — idempotence demonstrated against live hosts, in check mode only. `--list-hosts` confirms `-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take their OWN var; the wrong flag silently targets dev). *** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was deliberately left for the operator to schedule: prod's Longhorn had only just finished resyncing after a node reboot, and changing a node's clock config during storage recovery is not an unattended move. `ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values (the passphrase is mounted into the dev container) and has leaked the prod become-password to a transcript once already. === 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) === On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died, and `resume` would have enabled velero on prod against a nonexistent hostname: installs clean, reports healthy, no backups exist. Found by accident. A guard in one file keyed on the SHAPE of a value in another file, with nothing tying them. The check exercises the REAL predicate — it does not reimplement it. standup.sh is not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the check extracts the predicate's own source lines and sources only those. Faithfulness is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name), exactly-one-definition, "extracted block is a verbatim substring", a self-containment gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent second copy. If the predicate grows a dependency the check FAILS LOUDLY and names the remedy rather than degrading to testing a copy — the failure mode it exists for. Both sides derive from real files: guarded knobs from a taint trace in standup.sh (`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent detector is deliberately NOT the guard: seed idioms UNION the guard's own words with glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm cannot make a value stop being placeholder-shaped. Real tree: 32 assertions, 18 values through the extracted predicate, PASS. NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the read-only spec repo untouched). My first perturbation tripped the self-containment rail instead of the anchoring property, so I redid it shape-preserving — substring globs -> prefix globs, nothing else — and it failed on exactly the right property, naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05 failure exactly". Control: unperturbed copy exits 0. === 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) === The path->requirement map had no `bootstrap/*` entry, so a change touching standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's requirements and silently excluded the new check — the one that exists to catch exactly the 2026-09-05 class. A safety net with a hole where the incident was. Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths. VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check. Coverage unchanged at 45/51 (the new check tags existing requirements). Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the bootstrap/ entrypoints.
LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.
Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
Prerequisites
tofu applyin../tofu/has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).- Inventory reflects the node IPs.
inventory/hosts.iniships aligned with the LOCKED defaults; regenerate from tofu if overridden:cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini - Run from the dev container (has
kubectl+helm, used by the Cilium/verify/ Flux plays viadelegate_to: localhost).
Run order
cd infra/ansible
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
site.ymlis NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDEsite.ymland must be run separately:ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3(Both take the same target-group override as the rest — see Targeting prod below.)
Standalone diagnostics (not part of any bring-up; safe to point anywhere):
ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
memattr-survey.ymlis the only play here that reboots its targets, and it will not do so until-e memattr_survey_reboots_authorised=trueis passed. Seeroles/memattr_survey/README.mdandinfra/PROD-PROVISIONING.md§0b.
Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes everything on the host filesystem — including host-level config
that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the
node-host layers or the node comes back missing pieces:
| Re-run | Why it doesn't survive a rebuild |
|---|---|
node-baseline.yml |
packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off |
k3s.yml |
the k3s server binary + config + join (secondaries re-join the VIP) |
kube-vip.yml |
the kube-vip auto-deploy manifest on the server's host path |
vpn.egress.gateway.yml |
the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet |
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip
should return a PIA exit IP (not the home ISP).
What each layer does
| Playbook / role | Action |
|---|---|
k3s.yml / k3s_server |
k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored) |
cilium.yml / cilium |
Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements |
verify.yml / cluster_verify |
waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
flux-bootstrap.yml / flux_bootstrap |
calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev) |
Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config
afterward. The parity-critical values must match between this role's
templates/cilium-values.yaml.j2 and that HelmRelease:
kubeProxyReplacement: truek8sServiceHost: {{ k3s_api_host }}(= primary node IP,10.73.0.11by default)k8sServicePort: 6443
The HelmRelease currently has k8sServiceHost: "REPLACE-ME" — WS-NET must set it
to the same value this role uses (surfaced by tofu output primary_node_ip).
Interface assumed of bootstrap/ (authored by other workers, NOT here)
roles/flux_bootstrap invokes two entrypoints via overridable command vars:
flux_bootstrap_sops_cmd→bootstrap/sops/install-age-key.sh(Security Engineer, T-1.4): installs the age private key as thesops-ageSecret influx-system. Key never committed (.gitignoreguardsage.key/*.key).flux_bootstrap_flux_cmd→bootstrap/flux/bootstrap.sh(GitOps Engineer, T-1.4): runsflux bootstrap ... --path=clusters/dev.
These script names/args are assumptions — if the bootstrap deliverables
expose different entrypoints, override the two vars in
roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged
for PM reconciliation.
Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
inventory/hosts.ini carries a [k3s_prod_primary] / [k3s_prod_secondary] /
[k3s_prod:children] section for the three planned bare-metal nodes (k3s-prod-1 at
192.168.2.10 exists and is reachable today; k3s-prod-2/-3 at .11/.12 are addressed but
not yet racked, so their inventory lines stay commented out), with its own group_vars/k3s_prod/
(vaulted ansible_become_password + non-secret overrides).
Nothing here is forked for prod. Every node-host play takes its target group from a variable that defaults to the dev group, so the dev commands above are unchanged, and prod is the same command plus an override. Two separate mechanisms, for two separate reasons:
| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play runs on | k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts, vpn_egress_gateway_hosts |
-e extra vars only |
Ansible templates a play's hosts: before it has any host, so it cannot see group_vars at all |
| Which group a role means by "my primary" / "my servers" | k3s_primary_group, k3s_servers_group (+ cluster_kubeconfig_name) |
group_vars — dev in all.yml, prod in k3s_prod/vars.yml |
Host context exists by then, so each environment answers for itself with no per-run flag — same mechanism k3s_api_host already used |
playbooks/node-health.yml established the first pattern (-e node_health_hosts=k3s_prod); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
playbooks/memattr-survey.yml follows it (-e memattr_survey_hosts=…, defaulting to k3s_prod)
for the same reason, and adds a second interlock on top: because it reboots its targets, the
role also refuses any host that is not a member of memattr_survey_required_group. A host pattern
can be mistyped; inventory membership cannot.
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml
# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
-e k3s_hosts=k3s_prod \
-e k3s_primary_hosts=k3s_prod_primary \
-e k3s_secondary_hosts=k3s_prod_secondary
What the roles then resolve by themselves for a prod host, from group_vars/k3s_prod/vars.yml:
k3s_primary_group=k3s_prod_primary (the --cluster-init founder, the node-token delegate_to
target, and the node Cilium fetches the kubeconfig from), k3s_servers_group=k3s_prod (the tls-san
node-IP list), k3s_api_host=192.168.2.10, kube_vip_address=192.168.2.9, and
cluster_kubeconfig_name=prod.kubeconfig — so a prod run can never overwrite the
.cache/dev.kubeconfig that infra/devcontainer/dev.sh exports as $KUBECONFIG.
Packages-only baseline (--tags packages)
roles/node_baseline is tagged packages / system / disks. An untagged run is unchanged (all
three, in order). --tags packages runs only the apt task:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
This exists to break a circular dependency: prod's disk layout (infra/PROD-PROVISIONING.md §2) is
blocked on a SMART read, SMART needs smartmontools, smartmontools ships in
node_baseline_packages — but the role's disk tasks need block devices that decision hasn't chosen
yet. With --tags packages no disks-tagged task is selected, so the deliberate
/dev/BLOCKED-pending-disk-decision-… placeholders in group_vars/k3s_prod/vars.yml are never
even templated, let alone handed to mkfs. A full run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
Still not runnable end to end
- The disk-layout (§2) and the
node_baseline_*_deviceoverrides remain BLOCKED placeholders. group_vars/k3s_prod/vault.yml(the become password) does not exist yet, so no prod play can escalate — and--checkagainst prod fails for the same reason.flux-bootstrap.ymlretargets, butroles/flux_bootstrapshells out tobootstrap/flux/bootstrap.sh, which bootstraps--path=clusters/dev. Prod needs--path=clusters/prod: abootstrap/change, not a target-group change (§7).vpn.egress.gateway.ymlis parameterised (-e vpn_egress_gateway_hosts=…) but prod has no egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own vaulted PIA credentials.- The operator has ruled the prod cluster stays uninitialized until all three nodes are present
(§8).
192.168.2.20is prod's ingress Gateway VIP — a Flux/cluster-varsconcern, not an Ansible one; the kube-vip control-plane VIP is192.168.2.9(kube_vip_address).
Everything above this section describes the dev runbook; the values in it (192.168.1.x,
.cache/dev.kubeconfig, clusters/dev) are dev's, supplied by group_vars, not baked into the
roles.
Static validation (PM, in dev container — NOT run here)
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/