1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
forwarding on dev cannot recur. Prod deliberately differs from dev, which is
PiHole-first because dev needs the local jackson.lan zone.
SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
only one is configured here:
pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
PiHole is NOT in this path. Prod is already correct (.20).
LAN -> PiHole. Not a cluster concern.
nodes -> THIS FILE.
The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
recorded in PROD-PROVISIONING.md §4 for the operator to pick.
2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
not DR for prod.
3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
--become` returns root. The L0 no-plaintext-secrets check asserts every ansible
vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.
4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
sets collections_path = collections, and the collections are installed there from the
existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
setup, vendored bytes are not source).
5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
losses on a consumer drive with NO power-loss protection, in a single failure domain.
etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
PLP is exactly how a member corrupts.
6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
one node is outweighed by the other two and causes node ejection. That holds for the
dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
faithfully to all three replicas, so quorum sees no disagreement — low probability, and
the only mitigation is different hardware. Follow-up noted: Longhorn's
snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
is the mechanism that operationalises the operator's own argument.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.
Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.
smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.
FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.
Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.
Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.
Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.
NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.
BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.
Serves: R-ENV-03, R-OBS-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ask: implement the roles needed to check prod node hardware health, and run
them to find out our state. This is that role; the run's findings are below.
roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware
health), styled after node_baseline. One unprivileged collector script does the work in
a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq +
thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk,
lscpu, mounts + statvfs, per-NIC stats, resolv.conf.
THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never
as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes"
section. A fake zero on a reallocated-sector count is exactly the kind of false green
this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why.
Report-only by default; thresholds live in defaults/main.yml.
Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root)
and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip
cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live
replica is never mistaken for a spare device.
playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts=
k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the
group-name gap already flagged in hosts.ini.
Two become facts found the hard way and documented in the role: a group_vars
`ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged
tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo
password" — only `ignore_errors` does.
FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO.
Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero
throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at
1000 Mb/s, BIOS R21 02.26.00, no reboot pending.
DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no
vaulted ansible_become_password for prod, AND smartmontools is not on the node at all
(node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors
and drive temperature all remain unmeasured on a ~2011 consumer SSD.
WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on
dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media
trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does
the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which
refines rather than resolves the DNS-SPOF finding.
ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc
exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a
prod node would be undetectable. Recorded as a SKIP with reason, not a pass.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.
BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.
PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.
ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.
BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.
KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.
Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A node rebuild (tofu -replace / fresh OS) wipes host-level config Flux/Helm don't manage.
site.yml omits kube-vip.yml and vpn.egress.gateway.yml, so a re-provision missed the node-host
OpenVPN on k3s-dev-3 and dropped all egress-class=vpn pod traffic. Document the node-host layers
that must be re-run after any re-provision, and correct the stale author-only banner (these plays
are live).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across
0.8->1.2, so the bump is image + re-applied auto-deploy manifest.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, leader/primary last. variables.tf image defaults -> trixie generic.
k3s_server role: ensure /etc/rancher/k3s exists (fresh-node re-provision fix).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gateway API CRDs bumped v1.2.1->v1.3.0 (required by Cilium 1.18; applied server-side).
Land on current k8s 1.32 before the 1.33 hop.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop.
Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs
stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.
tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
(honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
dropped the cross-type moved{} linkage to the short proxmox_download_file, so
the stale moved block failed EVERY plan with "Resource type mismatch" (broken
on main too). Aligning code to state yields a clean plan with no image
re-download. Short-name migration is deferred to a deliberate `tofu state mv`.
ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
/var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
disk fails loudly rather than silently landing etcd on the root HDD) before
k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
filesystem module never reformats an existing fs.
Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.
Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
- decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
- installs the pinned, checksummed versitygw binary (fetched+extracted on the
controller, host has no internet), a dedicated versitygw system user, and a
Type=simple hardened systemd unit running the posix backend rooted at
/mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
plain HTTP / path-style
- ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
- pre-creates the velero + cnpg buckets (bucket == top-level subdir)
Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).
Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the
off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05).
Ansible (author + executed):
- roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only:
ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no
sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the
controller since the host has no WAN, copied to host); hardened systemd unit bound to
the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets;
a non-root access key scoped by IAM policy to only those two buckets.
- inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide
ansible_user. Does not disturb existing k3s node entries.
- group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds.
GitOps hand-off:
- platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint +
scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a
kustomization (left for the consuming agents).
Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000;
S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod;
buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lets users kubectl into the cluster via Keycloak SSO, mapping realm groups to
cluster RBAC. The cert-based admin kubeconfig is untouched and remains a fully
independent break-glass path (works even if Keycloak/OIDC is down).
- keycloak-realm-import.yaml: new `kubernetes` PUBLIC client + PKCE (S256) — no
client secret in any kubeconfig; kubelogin runs auth-code+PKCE to localhost.
groups mapper + email/profile/roles scopes.
- k8s-oidc-rbac.yaml (NEW, platform-config): ClusterRoleBindings
oidc:platform-admins -> cluster-admin, oidc:viewers -> view.
- k3s_server role: kube-apiserver-arg OIDC flags (issuer=homelab realm,
client-id=kubernetes, username-claim=email + oidc: prefix, groups-claim=groups
+ oidc: prefix). No oidc-ca-file (nodes reach the issuer via the gateway VIP
and trust the public LE cert). Applied via the rolling Restart k3s handler.
Keycloak/RBAC land via Flux now; the API-server flags take effect on the next
`ansible-playbook playbooks/k3s.yml` rolling restart (deferred until cluster load
from the VM-import incident settles).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The k3s nodes authorize ~/.ssh/home_network for the `k3s` user, but ansible.cfg
set no private_key_file and there's no ssh_config entry for the SSH-NAT host, so
node-config plays (e.g. the rolling k3s config.yaml + Restart handler) failed
with Permission denied (publickey). Pin the key so those plays connect without a
per-run --private-key.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi VM boot-disk DataVolume (RWX + Block on longhorn-vm) failed to import:
CDI's non-root importer (UID 107, cap-drop ALL) hit
`blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
Kubernetes only applies fsGroup ownership to *filesystem* volumes — it never
chowns a raw block device — and Longhorn's CSIDriver (fsGroupPolicy:
ReadWriteOnceWithFSType) skips fsGroup on RWX/Block volumes regardless, so the
device stays root:root. The CDI-documented fix is the container-runtime setting
`device_ownership_from_security_context`, exposed on k3s (>= v1.31.3+k3s1, this
cluster is v1.31.5) as `nonroot-devices: true`. It makes containerd set the
in-container device node's owner from the pod's securityContext.
Added to the Ansible-managed /etc/rancher/k3s/config.yaml alongside tls-san;
applied via the existing rolling `Restart k3s` handler (serial:1, kube-vip keeps
the API up, etcd holds 2/3). Node-scoped, security-neutral (pods stay non-root),
and preserves the longhorn-vm RWX/Block/migratable contract unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Controller now reaches the kube-API directly through the pfSense NAT
(192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS
against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel
scaffolding and updates the ansible source-of-truth so a rebuild
regenerates a direct-access kubeconfig.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add
the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the
VIP validates. Prepares the stable failover endpoint the operator fronts with
a pfSense port-forward to 192.168.1.49:6443.
roles/kube_vip + playbooks/kube-vip.yml:
- Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's
k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is
reproducible and independent of Flux; k3s auto-applies it.
- Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership
of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict.
- Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane),
hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9.
- vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0).
- Leader election (plndr-cp-lock) with 15/10/2s lease timings.
- This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork
pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the
node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader
election works.
roles/k3s_server tls-san:
- New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install
args now emit one --tls-san per entry.
- Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s)
and Restart k3s handler so an existing cluster regenerates its serving cert.
- k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time,
keeping etcd quorum (kube-vip keeps the API reachable during each restart).
Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and
an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all
nodes Ready throughout the rolling restart.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries
each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity
- dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted
playbooks can run in the pinned container (ansible.cfg vault_password_file)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- ansible hosts.ini: node ansible_host -> 192.168.1.50/.51/.52, with a
clear note that the PM may need to swap these for the pfSense
NAT-forwarded address:port once that NAT rule exists (not guessed here).
k3s_api_host derives from the inventory, so it follows automatically.
- Cilium CiliumLoadBalancerIPPool -> 192.168.1.53-.60 (matches the tofu
cilium_l2_vip reservation on the new LAN).
- CiliumL2AnnouncementPolicy: broaden interface selector to ^e(n|th) so
the Proxmox NIC name (ens18/eth0) need not be pinned; flagged
verify-at-bringup.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The node-tunnel PIA credentials (username/password/.ovpn) live host-side in Ansible vault per
docs/07. The encrypted vault.yml was provisioned but never committed, so the vpn.egress.gateway
playbook was not reproducible from a fresh checkout. Commit it (ansible-vault encrypted; the
vault password stays out of the repo via ansible.cfg vault_password_file).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The down-script previously flushed the kill-switch source set on tunnel-down, which
disarmed the 'ip saddr @vpn_egress_gateway_src oifname != tun0 drop' rule exactly when
the tunnel was down — leaving fail-closed to ride only on the stale SNAT source being
non-routable (could leak if the edge NATed it). Retain the set instead: the drop rule
stays armed so any vpn-class packet leaving the WAN while the tunnel is down is EXPLICITLY
dropped (edge-independent). Safe because the up-script flush+re-adds the source on every
reconnect. Validated behavior: WS-NET fault-injection on k3s-dev-3 (2026-08-24).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).
Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
redirect-gateway => installs no routes; policy routing in a dedicated table
(51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
never matched.
PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The deployed cluster has TWO default StorageClasses — k3s's built-in local-path
AND longhorn — which is ambiguous for PVCs without an explicit class. Longhorn
(ADR-0003) is the cluster storage, so add local-storage to k3s --disable. Effective
on fresh installs (the k3s install task is idempotent → no restart of the running
cluster); the running dev cluster keeps both defaults harmlessly (all platform PVCs
pin storageClassName: longhorn). Rebuilds get a single default.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements T-1.5 findings F1/F2 (author-first, no cluster change).
F1 (tofu): refactor libvirt node resources count -> for_each keyed by
node name so single-node -replace/destroy cannot cascade to siblings.
Add ordered-list/primary helper locals; update outputs.tf off the map;
add moved.tf (12 moved blocks). tofu plan proves a pure state-address
migration: 0 to add, 0 to change, 0 to destroy (all 12 'has moved to').
State-persist via a 0-change apply is operator-permission-gated.
F2 (ansible): k3s_server secondary fetches the join token via
delegate_to the primary (published fact as fast path), so a single-node
rejoin with --limit k3s-dev-N is self-contained and no longer depends on
a fact an earlier play had to set. Syntax-check + CI gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
host-prereqs.yml opened the firewall (via the operator's ansible-vault become
password); Flux GitRepository READY=True (main@sha1:a2f3bdc). Commit the
encrypted hypervisor vault (ansible_become_password) so rebuild is fully
hands-off; passphrase stays out-of-repo. platform/apps still not reconciled.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Where the sudo password goes, codified:
- inventory: new [hypervisor] group (localhost, connection=local).
- host-prereqs.yml targets 'hypervisor'; become password from vaulted
group_vars/hypervisor/vault.yml (ansible_become_password) — user creates it
encrypted (safe to commit); README documents the exact commands.
- ansible.cfg vault_password_file = ~/.config/home-k3s/ansible-vault-pass
(passphrase kept OUTSIDE the repo, never committed).
- ci/yamllint.yaml ignores **/vault.yml (vault blob isn't parseable YAML).
Nodes keep NOPASSWD sudo (group_vars/all) — vault scoped to the host only.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per operator: the firewalld rule (VM->host:3000 for the Flux<-Gitea source) must be
IaC, not a manual command (R-ENV-01, R-PLAT-04).
- infra/ansible/playbooks/host-prereqs.yml: idempotent firewalld open on the libvirt
zone (localhost play, become; no extra collection).
- bootstrap/gitea/bootstrap-gitea.sh: now fully automates admin + public repo + push
(was manual instructions) — idempotent; creds stay 0600 on host, never committed.
Applying host-prereqs still needs host sudo (I lack non-interactive sudo); with
passwordless become a destroy/rebuild is fully unattended.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
