The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.
Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.
smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.
FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.
Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.
Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.
Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.
NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.
BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.
Serves: R-ENV-03, R-OBS-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ask: implement the roles needed to check prod node hardware health, and run
them to find out our state. This is that role; the run's findings are below.
roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware
health), styled after node_baseline. One unprivileged collector script does the work in
a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq +
thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk,
lscpu, mounts + statvfs, per-NIC stats, resolv.conf.
THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never
as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes"
section. A fake zero on a reallocated-sector count is exactly the kind of false green
this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why.
Report-only by default; thresholds live in defaults/main.yml.
Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root)
and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip
cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live
replica is never mistaken for a spare device.
playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts=
k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the
group-name gap already flagged in hosts.ini.
Two become facts found the hard way and documented in the role: a group_vars
`ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged
tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo
password" — only `ignore_errors` does.
FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO.
Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero
throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at
1000 Mb/s, BIOS R21 02.26.00, no reboot pending.
DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no
vaulted ansible_become_password for prod, AND smartmontools is not on the node at all
(node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors
and drive temperature all remain unmeasured on a ~2011 consumer SSD.
WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on
dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media
trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does
the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which
refines rather than resolves the DNS-SPOF finding.
ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc
exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a
prod node would be undetectable. Recorded as a SKIP with reason, not a pass.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.
BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.
PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.
ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.
BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.
KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.
Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across
0.8->1.2, so the bump is image + re-applied auto-deploy manifest.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, leader/primary last. variables.tf image defaults -> trixie generic.
k3s_server role: ensure /etc/rancher/k3s exists (fresh-node re-provision fix).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gateway API CRDs bumped v1.2.1->v1.3.0 (required by Cilium 1.18; applied server-side).
Land on current k8s 1.32 before the 1.33 hop.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop.
Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs
stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.
tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
(honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
dropped the cross-type moved{} linkage to the short proxmox_download_file, so
the stale moved block failed EVERY plan with "Resource type mismatch" (broken
on main too). Aligning code to state yields a clean plan with no image
re-download. Short-name migration is deferred to a deliberate `tofu state mv`.
ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
/var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
disk fails loudly rather than silently landing etcd on the root HDD) before
k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
filesystem module never reformats an existing fs.
Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.
Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
- decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
- installs the pinned, checksummed versitygw binary (fetched+extracted on the
controller, host has no internet), a dedicated versitygw system user, and a
Type=simple hardened systemd unit running the posix backend rooted at
/mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
plain HTTP / path-style
- ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
- pre-creates the velero + cnpg buckets (bucket == top-level subdir)
Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).
Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the
off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05).
Ansible (author + executed):
- roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only:
ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no
sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the
controller since the host has no WAN, copied to host); hardened systemd unit bound to
the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets;
a non-root access key scoped by IAM policy to only those two buckets.
- inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide
ansible_user. Does not disturb existing k3s node entries.
- group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds.
GitOps hand-off:
- platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint +
scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a
kustomization (left for the consuming agents).
Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000;
S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod;
buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lets users kubectl into the cluster via Keycloak SSO, mapping realm groups to
cluster RBAC. The cert-based admin kubeconfig is untouched and remains a fully
independent break-glass path (works even if Keycloak/OIDC is down).
- keycloak-realm-import.yaml: new `kubernetes` PUBLIC client + PKCE (S256) — no
client secret in any kubeconfig; kubelogin runs auth-code+PKCE to localhost.
groups mapper + email/profile/roles scopes.
- k8s-oidc-rbac.yaml (NEW, platform-config): ClusterRoleBindings
oidc:platform-admins -> cluster-admin, oidc:viewers -> view.
- k3s_server role: kube-apiserver-arg OIDC flags (issuer=homelab realm,
client-id=kubernetes, username-claim=email + oidc: prefix, groups-claim=groups
+ oidc: prefix). No oidc-ca-file (nodes reach the issuer via the gateway VIP
and trust the public LE cert). Applied via the rolling Restart k3s handler.
Keycloak/RBAC land via Flux now; the API-server flags take effect on the next
`ansible-playbook playbooks/k3s.yml` rolling restart (deferred until cluster load
from the VM-import incident settles).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi VM boot-disk DataVolume (RWX + Block on longhorn-vm) failed to import:
CDI's non-root importer (UID 107, cap-drop ALL) hit
`blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
Kubernetes only applies fsGroup ownership to *filesystem* volumes — it never
chowns a raw block device — and Longhorn's CSIDriver (fsGroupPolicy:
ReadWriteOnceWithFSType) skips fsGroup on RWX/Block volumes regardless, so the
device stays root:root. The CDI-documented fix is the container-runtime setting
`device_ownership_from_security_context`, exposed on k3s (>= v1.31.3+k3s1, this
cluster is v1.31.5) as `nonroot-devices: true`. It makes containerd set the
in-container device node's owner from the pod's securityContext.
Added to the Ansible-managed /etc/rancher/k3s/config.yaml alongside tls-san;
applied via the existing rolling `Restart k3s` handler (serial:1, kube-vip keeps
the API up, etcd holds 2/3). Node-scoped, security-neutral (pods stay non-root),
and preserves the longhorn-vm RWX/Block/migratable contract unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Controller now reaches the kube-API directly through the pfSense NAT
(192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS
against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel
scaffolding and updates the ansible source-of-truth so a rebuild
regenerates a direct-access kubeconfig.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add
the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the
VIP validates. Prepares the stable failover endpoint the operator fronts with
a pfSense port-forward to 192.168.1.49:6443.
roles/kube_vip + playbooks/kube-vip.yml:
- Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's
k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is
reproducible and independent of Flux; k3s auto-applies it.
- Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership
of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict.
- Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane),
hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9.
- vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0).
- Leader election (plndr-cp-lock) with 15/10/2s lease timings.
- This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork
pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the
node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader
election works.
roles/k3s_server tls-san:
- New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install
args now emit one --tls-san per entry.
- Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s)
and Restart k3s handler so an existing cluster regenerates its serving cert.
- k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time,
keeping etcd quorum (kube-vip keeps the API reachable during each restart).
Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and
an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all
nodes Ready throughout the rolling restart.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries
each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity
- dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted
playbooks can run in the pinned container (ansible.cfg vault_password_file)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The down-script previously flushed the kill-switch source set on tunnel-down, which
disarmed the 'ip saddr @vpn_egress_gateway_src oifname != tun0 drop' rule exactly when
the tunnel was down — leaving fail-closed to ride only on the stale SNAT source being
non-routable (could leak if the edge NATed it). Retain the set instead: the drop rule
stays armed so any vpn-class packet leaving the WAN while the tunnel is down is EXPLICITLY
dropped (edge-independent). Safe because the up-script flush+re-adds the source on every
reconnect. Validated behavior: WS-NET fault-injection on k3s-dev-3 (2026-08-24).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).
Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
redirect-gateway => installs no routes; policy routing in a dedicated table
(51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
never matched.
PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The deployed cluster has TWO default StorageClasses — k3s's built-in local-path
AND longhorn — which is ambiguous for PVCs without an explicit class. Longhorn
(ADR-0003) is the cluster storage, so add local-storage to k3s --disable. Effective
on fresh installs (the k3s install task is idempotent → no restart of the running
cluster); the running dev cluster keeps both defaults harmlessly (all platform PVCs
pin storageClassName: longhorn). Rebuilds get a single default.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements T-1.5 findings F1/F2 (author-first, no cluster change).
F1 (tofu): refactor libvirt node resources count -> for_each keyed by
node name so single-node -replace/destroy cannot cascade to siblings.
Add ordered-list/primary helper locals; update outputs.tf off the map;
add moved.tf (12 moved blocks). tofu plan proves a pure state-address
migration: 0 to add, 0 to change, 0 to destroy (all 12 'has moved to').
State-persist via a 0-change apply is operator-permission-gated.
F2 (ansible): k3s_server secondary fetches the join token via
delegate_to the primary (published fact as fast path), so a single-node
rejoin with --limit k3s-dev-N is self-contained and no longer depends on
a fact an earlier play had to set. Syntax-check + CI gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
