feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path

1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS forwarding on dev cannot recur. Prod deliberately differs from dev, which is PiHole-first because dev needs the local jackson.lan zone. SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and only one is configured here: pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster; PiHole is NOT in this path. Prod is already correct (.20). LAN -> PiHole. Not a cluster concern. nodes -> THIS FILE. The one thing public node DNS breaks: the k3s API SERVER is a node process carrying oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the issuer through the NODE resolver, so prod would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are recorded in PROD-PROVISIONING.md §4 for the operator to pick. 2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is not DR for prod. 3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un' --become` returns root. The L0 no-plaintext-secrets check asserts every ansible vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate. 4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now sets collections_path = collections, and the collections are installed there from the existing collections/requirements.yml (gitignored — pinned by requirements, fetched at setup, vendored bytes are not source). 5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power losses on a consumer drive with NO power-loss protection, in a single failure domain. etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without PLP is exactly how a member corrupts. 6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on one node is outweighed by the other two and causes node ejection. That holds for the dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity + rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates faithfully to all three replicas, so quorum sees no disagreement — low probability, and the only mitigation is different hardware. Follow-up noted: Longhorn's snapshotDataIntegrity is left at the chart default rather than set explicitly, and it is the mechanism that operationalises the operator's own argument. Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 00:40:29 -05:00
1 parent 0b3db6e778
commit ff0533106b
@@ -1482,7 +1482,17 @@ thing that ever closes a requirement.
 - `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare
  tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is
  unambiguous.
- **ECC is unknowable on this hardware** — no EDAC controller bound; non-ECC consumer RAM.
-  Silent memory corruption on a prod node would be invisible. Hardware property, not fixable.
+- **Non-ECC RAM — RISK ACCEPTED by the operator (2026-09-01), not an open item.** `node_health`
+  correctly reports ECC status as unknowable (no EDAC controller bound). Operator rationale:
+  this is a 3-node k3s homelab, not enterprise k8s; corruption on one node is outweighed by the
+  other two and results in node ejection. That holds for the dominant failure mode — etcd raft
+  (per-entry CRC + boltdb page checksums, quorum of 2 survives) and Longhorn (3 replicas, hard
+  anti-affinity, rebuild from healthy peers) both detect a *divergent* node and eject/rebuild it.
+  The residual it does not cover is corruption introduced *upstream* of replication: bytes
+  corrupted before the checksum is computed replicate faithfully to all three replicas, so
+  quorum sees no disagreement. Low probability, and the mitigation (ECC hardware) is out of
+  scope for this platform. **Do not re-litigate.** The one thing that operationalises the
+  operator's own argument is Longhorn's `snapshotDataIntegrity`, which is currently left at the
+  chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
 - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
  the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
@@ -103,10 +103,10 @@ patches:
      # collide with the Gateway VIP or the node IPs (.10/.11/.12).
      - op: replace
        path: /data/UNIFI_L4_VIP
-        value: "TBD-PROD-UNIFI-L4-VIP"
+        value: "192.168.2.21"
      - op: replace
        path: /data/CRAFTY_L4_VIP
-        value: "TBD-PROD-CRAFTY-L4-VIP"
+        value: "192.168.2.22"
      # Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
      # network — a prod backup written there is not off-cluster DR and would mix two environments'
      # PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
@@ -69,9 +69,38 @@ sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)

 Consequences to settle before imaging nodes 2 and 3:

- [ ] **Drive health checked first** (operator decision, 2026-08-31). See §5 for the command —
-      needs root. A ~15-year-old consumer SSD is not an obvious place for prod state.
- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
+- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
+      floor — see §5). Raw attributes, so the numbers are auditable:
+
+      | | |
+      |---|---|
+      | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
+      | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
+      | Power-on hours | 6,814 (~284 days) · 147 power cycles |
+      | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
+      | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
+      | **Erase failures** | **187** (`172`) |
+      | **Unexpected power losses** | **32** (`174`) |
+      | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
+      | UDMA CRC errors | 0 (`199`) — cabling is clean |
+
+      **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
+      count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
+      normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
+      not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
+      the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
+      uncorrectable counters instead.
+
+      **Verdict: usable, but not for production state.** Nothing says imminent failure — no
+      pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
+      consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
+      blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
+      power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
+      it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
+      a member corrupts — this drive has already logged 32 of them.
+      (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
+      `DECISIONS-LOG.md`. It is not part of this verdict.)
+- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
      (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
      Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
      prod must override them in `group_vars`. On a single-disk node there is no second device at
@@ -6,3 +6,8 @@
 # Regenerated on every run and they carry drive serial numbers — never commit. Same
 # convention as conformance/_artifacts/: generated output is an artifact, not source.
 _artifacts/
+
+# Galaxy collections installed by `ansible-galaxy collection install -r
+# collections/requirements.yml -p collections/`. Pinned by requirements.yml, fetched at
+# setup time — vendored bytes are not source. ansible.cfg sets collections_path.
+collections/ansible_collections/
@@ -1,6 +1,10 @@
 [defaults]
 inventory = inventory/hosts.ini
 roles_path = roles
+# Galaxy collections live in the repo (gitignored), installed from
+# collections/requirements.yml. The devcontainer image does NOT ship them, so without
+# this every play dies at parse time on community.general.modprobe.
+collections_path = collections
 host_key_checking = False
 retry_files_enabled = False
 # SSH key the k3s nodes authorize for the `k3s` user (see inventory ansible_user).
@@ -140,9 +140,33 @@ node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIO
 # the file with exactly this list, so if they are to be kept they must be listed here
 # explicitly — as part of that same recorded decision.
 node_baseline_dns_servers:
-  - "192.168.2.1"
-  - "BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4"
-# Prod's local search domain is not a decided thing (dev's is jackson.lan, PiHole's zone).
-# An EMPTY list is the role's supported way to say "configure no search domain", which is
-# node 1's current behaviour — the conservative non-decision, not a missing value.
+  # DECIDED (operator, 2026-09-01): prod's resolvers are PUBLIC and deliberately differ
+  # from dev's. Dev is PiHole-first (192.168.1.18) because dev needs the local jackson.lan
+  # zone AND split-horizon *.home.keithjackson.dev. Prod has no local resolver on its LAN,
+  # so it takes two INDEPENDENT public upstreams — which resolves the single-upstream SPOF
+  # (a single upstream broke containerd image pulls and CoreDNS forwarding on dev).
+  - "8.8.8.8"
+  - "1.1.1.1"
+  # SCOPE — this file sets NODE resolution only: containerd image pulls, and the upstream
+  # CoreDNS forwards `.` to. Three different resolvers answer *.home.keithjackson.dev, and
+  # only ONE of them is configured here:
+  #
+  #   pods (in-cluster) -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template`
+  #       plugin answers the WHOLE home.keithjackson.dev zone with ${GATEWAY_VIP}. Entirely
+  #       self-contained in the cluster; PiHole is NOT in this path. Prod is already correct
+  #       (GATEWAY_VIP = 192.168.2.20 in clusters/prod).
+  #   LAN clients       -> PiHole. Not a cluster concern.
+  #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
+  #
+  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
+  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
+  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
+  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
+  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
+  # with the dev Gateway VIP.
+  #
+  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
+  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
+  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
+  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
 node_baseline_dns_search: []
@@ -0,0 +1,7 @@
+$ANSIBLE_VAULT;1.1;AES256
+33643331386138666338353162353562613737343265376639646262663931663239333735366163
+3961623239313038313966326339336464383763616266310a313766653231636636393030306635
+37653764656235363738363535386539613938393839623462316134383631623464343930383133
+6433386532653564630a383036633266353965383565666639363733393531643162366265366537
+30313265343035653961666331663534313564333939383038306338666535316639333136363036
+6532313164613836663862303561646362303130316136306361