feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path

1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
   upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
   forwarding on dev cannot recur. Prod deliberately differs from dev, which is
   PiHole-first because dev needs the local jackson.lan zone.

   SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
   only one is configured here:
     pods  -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
              answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
              PiHole is NOT in this path. Prod is already correct (.20).
     LAN   -> PiHole. Not a cluster concern.
     nodes -> THIS FILE.
   The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
   oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
   issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
   prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
   resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
   recorded in PROD-PROVISIONING.md §4 for the operator to pick.

2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
   first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
   within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
   exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
   not DR for prod.

3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
   proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
   --become` returns root. The L0 no-plaintext-secrets check asserts every ansible
   vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.

4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
   EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
   sets collections_path = collections, and the collections are installed there from the
   existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
   setup, vendored bytes are not source).

5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
   are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
   rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
   losses on a consumer drive with NO power-loss protection, in a single failure domain.
   etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
   PLP is exactly how a member corrupts.

6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
   item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
   one node is outweighed by the other two and causes node ejection. That holds for the
   dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
   rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
   faithfully to all three replicas, so quorum sees no disagreement — low probability, and
   the only mitigation is different hardware. Follow-up noted: Longhorn's
   snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
   is the mechanism that operationalises the operator's own argument.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

DECISIONS-LOG.md

@@ -1482,7 +1482,17 @@ thing that ever closes a requirement.
 - `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare
   tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is
   unambiguous.
-- **ECC is unknowable on this hardware** — no EDAC controller bound; non-ECC consumer RAM.
-  Silent memory corruption on a prod node would be invisible. Hardware property, not fixable.
+- **Non-ECC RAM — RISK ACCEPTED by the operator (2026-09-01), not an open item.** `node_health`
+  correctly reports ECC status as unknowable (no EDAC controller bound). Operator rationale:
+  this is a 3-node k3s homelab, not enterprise k8s; corruption on one node is outweighed by the
+  other two and results in node ejection. That holds for the dominant failure mode — etcd raft
+  (per-entry CRC + boltdb page checksums, quorum of 2 survives) and Longhorn (3 replicas, hard
+  anti-affinity, rebuild from healthy peers) both detect a *divergent* node and eject/rebuild it.
+  The residual it does not cover is corruption introduced *upstream* of replication: bytes
+  corrupted before the checksum is computed replicate faithfully to all three replicas, so
+  quorum sees no disagreement. Low probability, and the mitigation (ECC hardware) is out of
+  scope for this platform. **Do not re-litigate.** The one thing that operationalises the
+  operator's own argument is Longhorn's `snapshotDataIntegrity`, which is currently left at the
+  chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
 - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
   the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.