home-network-k3s-impl

1,078 Commits 139 Branches 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	ff0533106b	feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details 1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS forwarding on dev cannot recur. Prod deliberately differs from dev, which is PiHole-first because dev needs the local jackson.lan zone. SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer .home.keithjackson.dev and only one is configured here: pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster; PiHole is NOT in this path. Prod is already correct (.20). LAN -> PiHole. Not a cluster concern. nodes -> THIS FILE. The one thing public node DNS breaks: the k3s API SERVER is a node process carrying oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the issuer through the NODE resolver, so prod would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are recorded in PROD-PROVISIONING.md §4 for the operator to pick. 2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD- : no prod backup target exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is not DR for prod. 3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un' --become` returns root. The L0 no-plaintext-secrets check asserts every ansible vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate. 4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now sets collections_path = collections, and the collections are installed there from the existing collections/requirements.yml (gitignored — pinned by requirements, fetched at setup, vendored bytes are not source). 5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power losses on a consumer drive with NO power-loss protection, in a single failure domain. etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without PLP is exactly how a member corrupts. 6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on one node is outweighed by the other two and causes node ejection. That holds for the dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity + rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates faithfully to all three replicas, so quorum sees no disagreement — low probability, and the only mitigation is different hardware. Follow-up noted: Longhorn's snapshotDataIntegrity is left at the chart default rather than set explicitly, and it is the mechanism that operationalises the operator's own argument. Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 00:40:29 -05:00
jakman85andClaude Opus 5	9f59c36000	feat(ansible): node_health role — read-only hardware health for prod + dev nodes Operator ask: implement the roles needed to check prod node hardware health, and run them to find out our state. This is that role; the run's findings are below. roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware health), styled after node_baseline. One unprivileged collector script does the work in a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq + thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk, lscpu, mounts + statvfs, per-NIC stats, resolv.conf. THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes" section. A fake zero on a reallocated-sector count is exactly the kind of false green this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why. Report-only by default; thresholds live in defaults/main.yml. Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root) and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live replica is never mistaken for a spare device. playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts= k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the group-name gap already flagged in hosts.ini. Two become facts found the hard way and documented in the role: a group_vars `ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo password" — only `ignore_errors` does. FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO. Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at 1000 Mb/s, BIOS R21 02.26.00, no reboot pending. DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no vaulted ansible_become_password for prod, AND smartmontools is not on the node at all (node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors and drive temperature all remain unmeasured on a ~2011 consumer SSD. WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which refines rather than resolves the DNS-SPOF finding. ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a prod node would be undetectable. Recorded as a SKIP with reason, not a pass. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 23:20:25 -05:00
jakman85andClaude Opus 4.8	f2a7e42068	Phase 1 (author-only): node provisioning IaC — tofu + cloud-init + Ansible (T-1.1/1.2/1.3) OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS + 40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate = Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-16 01:21:00 -05:00

Author

SHA1

Message

Date

jakman85andClaude Opus 5

ff0533106b

feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path

gate / gate (push) Has been cancelled

Details

gate / conformance (push) Has been cancelled

Details

1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
   upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
   forwarding on dev cannot recur. Prod deliberately differs from dev, which is
   PiHole-first because dev needs the local jackson.lan zone.

   SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
   only one is configured here:
     pods  -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
              answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
              PiHole is NOT in this path. Prod is already correct (.20).
     LAN   -> PiHole. Not a cluster concern.
     nodes -> THIS FILE.
   The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
   oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
   issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
   prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
   resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
   recorded in PROD-PROVISIONING.md §4 for the operator to pick.

2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
   first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
   within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
   exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
   not DR for prod.

3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
   proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
   --become` returns root. The L0 no-plaintext-secrets check asserts every ansible
   vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.

4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
   EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
   sets collections_path = collections, and the collections are installed there from the
   existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
   setup, vendored bytes are not source).

5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
   are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
   rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
   losses on a consumer drive with NO power-loss protection, in a single failure domain.
   etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
   PLP is exactly how a member corrupts.

6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
   item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
   one node is outweighed by the other two and causes node ejection. That holds for the
   dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
   rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
   faithfully to all three replicas, so quorum sees no disagreement — low probability, and
   the only mitigation is different hardware. Follow-up noted: Longhorn's
   snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
   is the mechanism that operationalises the operator's own argument.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-01 00:40:29 -05:00

jakman85andClaude Opus 5

9f59c36000

feat(ansible): node_health role — read-only hardware health for prod + dev nodes

Operator ask: implement the roles needed to check prod node hardware health, and run
them to find out our state. This is that role; the run's findings are below.

roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware
health), styled after node_baseline. One unprivileged collector script does the work in
a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq +
thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk,
lscpu, mounts + statvfs, per-NIC stats, resolv.conf.

THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never
as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes"
section. A fake zero on a reallocated-sector count is exactly the kind of false green
this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why.
Report-only by default; thresholds live in defaults/main.yml.

Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root)
and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip
cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live
replica is never mistaken for a spare device.

playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts=
k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the
group-name gap already flagged in hosts.ini.

Two become facts found the hard way and documented in the role: a group_vars
`ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged
tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo
password" — only `ignore_errors` does.

FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO.
Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero
throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at
1000 Mb/s, BIOS R21 02.26.00, no reboot pending.

DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no
vaulted ansible_become_password for prod, AND smartmontools is not on the node at all
(node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors
and drive temperature all remain unmeasured on a ~2011 consumer SSD.

WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on
dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media
trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does
the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which
refines rather than resolves the DNS-SPOF finding.

ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc
exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a
prod node would be undetectable. Recorded as a SKIP with reason, not a pass.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-08-31 23:20:25 -05:00

jakman85andClaude Opus 4.8

f2a7e42068

Phase 1 (author-only): node provisioning IaC — tofu + cloud-init + Ansible (T-1.1/1.2/1.3)

OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS +
40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn
prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway
API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a
cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate =
Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

2026-08-16 01:21:00 -05:00