diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 95f7348..63b347c 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1482,7 +1482,17 @@ thing that ever closes a requirement. - `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is unambiguous. -- **ECC is unknowable on this hardware** — no EDAC controller bound; non-ECC consumer RAM. - Silent memory corruption on a prod node would be invisible. Hardware property, not fixable. +- **Non-ECC RAM — RISK ACCEPTED by the operator (2026-09-01), not an open item.** `node_health` + correctly reports ECC status as unknowable (no EDAC controller bound). Operator rationale: + this is a 3-node k3s homelab, not enterprise k8s; corruption on one node is outweighed by the + other two and results in node ejection. That holds for the dominant failure mode — etcd raft + (per-entry CRC + boltdb page checksums, quorum of 2 survives) and Longhorn (3 replicas, hard + anti-affinity, rebuild from healthy peers) both detect a *divergent* node and eject/rebuild it. + The residual it does not cover is corruption introduced *upstream* of replication: bytes + corrupted before the checksum is computed replicate faithfully to all three replicas, so + quorum sees no disagreement. Low probability, and the mitigation (ECC hardware) is out of + scope for this platform. **Do not re-litigate.** The one thing that operationalises the + operator's own argument is Longhorn's `snapshotDataIntegrity`, which is currently left at the + chart default rather than set explicitly — see the follow-up in `platform/longhorn/`. - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI. diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index d7ddbf8..5b8b0f9 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -103,10 +103,10 @@ patches: # collide with the Gateway VIP or the node IPs (.10/.11/.12). - op: replace path: /data/UNIFI_L4_VIP - value: "TBD-PROD-UNIFI-L4-VIP" + value: "192.168.2.21" - op: replace path: /data/CRAFTY_L4_VIP - value: "TBD-PROD-CRAFTY-L4-VIP" + value: "192.168.2.22" # Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the # network — a prod backup written there is not off-cluster DR and would mix two environments' # PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port). diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index d24d82b..fa959a9 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -69,9 +69,38 @@ sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial) Consequences to settle before imaging nodes 2 and 3: -- [ ] **Drive health checked first** (operator decision, 2026-08-31). See §5 for the command — - needs root. A ~15-year-old consumer SSD is not an obvious place for prod state. -- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices* +- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package + floor — see §5). Raw attributes, so the numbers are auditable: + + | | | + |---|---| + | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) | + | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` | + | Power-on hours | 6,814 (~284 days) · 147 power cycles | + | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) | + | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) | + | **Erase failures** | **187** (`172`) | + | **Unexpected power losses** | **32** (`174`) | + | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) | + | UDMA CRC errors | 0 (`199`) — cabling is clean | + + **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector + count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its + normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**, + not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value; + the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent + uncorrectable counters instead. + + **Verdict: usable, but not for production state.** Nothing says imminent failure — no + pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011 + consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad + blocks, logged **187** erase failures, and taken **32** unclean power losses **with no + power-loss protection**, in a single failure domain. etcd is the worst possible tenant for + it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how + a member corrupts — this drive has already logged 32 of them. + (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see + `DECISIONS-LOG.md`. It is not part of this verdict.) +- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices* (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them. Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** — prod must override them in `group_vars`. On a single-disk node there is no second device at diff --git a/infra/ansible/.gitignore b/infra/ansible/.gitignore index 249fd8f..d7435a5 100644 --- a/infra/ansible/.gitignore +++ b/infra/ansible/.gitignore @@ -6,3 +6,8 @@ # Regenerated on every run and they carry drive serial numbers — never commit. Same # convention as conformance/_artifacts/: generated output is an artifact, not source. _artifacts/ + +# Galaxy collections installed by `ansible-galaxy collection install -r +# collections/requirements.yml -p collections/`. Pinned by requirements.yml, fetched at +# setup time — vendored bytes are not source. ansible.cfg sets collections_path. +collections/ansible_collections/ diff --git a/infra/ansible/ansible.cfg b/infra/ansible/ansible.cfg index 3667681..12cfe29 100644 --- a/infra/ansible/ansible.cfg +++ b/infra/ansible/ansible.cfg @@ -1,6 +1,10 @@ [defaults] inventory = inventory/hosts.ini roles_path = roles +# Galaxy collections live in the repo (gitignored), installed from +# collections/requirements.yml. The devcontainer image does NOT ship them, so without +# this every play dies at parse time on community.general.modprobe. +collections_path = collections host_key_checking = False retry_files_enabled = False # SSH key the k3s nodes authorize for the `k3s` user (see inventory ansible_user). diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 4071941..2113cf5 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -140,9 +140,33 @@ node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIO # the file with exactly this list, so if they are to be kept they must be listed here # explicitly — as part of that same recorded decision. node_baseline_dns_servers: - - "192.168.2.1" - - "BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4" -# Prod's local search domain is not a decided thing (dev's is jackson.lan, PiHole's zone). -# An EMPTY list is the role's supported way to say "configure no search domain", which is -# node 1's current behaviour — the conservative non-decision, not a missing value. + # DECIDED (operator, 2026-09-01): prod's resolvers are PUBLIC and deliberately differ + # from dev's. Dev is PiHole-first (192.168.1.18) because dev needs the local jackson.lan + # zone AND split-horizon *.home.keithjackson.dev. Prod has no local resolver on its LAN, + # so it takes two INDEPENDENT public upstreams — which resolves the single-upstream SPOF + # (a single upstream broke containerd image pulls and CoreDNS forwarding on dev). + - "8.8.8.8" + - "1.1.1.1" + # SCOPE — this file sets NODE resolution only: containerd image pulls, and the upstream + # CoreDNS forwards `.` to. Three different resolvers answer *.home.keithjackson.dev, and + # only ONE of them is configured here: + # + # pods (in-cluster) -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` + # plugin answers the WHOLE home.keithjackson.dev zone with ${GATEWAY_VIP}. Entirely + # self-contained in the cluster; PiHole is NOT in this path. Prod is already correct + # (GATEWAY_VIP = 192.168.2.20 in clusters/prod). + # LAN clients -> PiHole. Not a cluster concern. + # nodes -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY. + # + # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries + # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab` + # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on + # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl + # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone + # with the dev Gateway VIP. + # + # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in + # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning + # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the + # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC. node_baseline_dns_search: [] diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vault.yml b/infra/ansible/inventory/group_vars/k3s_prod/vault.yml new file mode 100644 index 0000000..1797329 --- /dev/null +++ b/infra/ansible/inventory/group_vars/k3s_prod/vault.yml @@ -0,0 +1,7 @@ +$ANSIBLE_VAULT;1.1;AES256 +33643331386138666338353162353562613737343265376639646262663931663239333735366163 +3961623239313038313966326339336464383763616266310a313766653231636636393030306635 +37653764656235363738363535386539613938393839623462316134383631623464343930383133 +6433386532653564630a383036633266353965383565666639363733393531643162366265366537 +30313265343035653961666331663534313564333939383038306338666535316639333136363036 +6532313164613836663862303561646362303130316136306361