jakman85andClaude Opus 5
ff0533106b
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
forwarding on dev cannot recur. Prod deliberately differs from dev, which is
PiHole-first because dev needs the local jackson.lan zone.
SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
only one is configured here:
pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
PiHole is NOT in this path. Prod is already correct (.20).
LAN -> PiHole. Not a cluster concern.
nodes -> THIS FILE.
The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
recorded in PROD-PROVISIONING.md §4 for the operator to pick.
2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
not DR for prod.
3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
--become` returns root. The L0 no-plaintext-secrets check asserts every ansible
vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.
4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
sets collections_path = collections, and the collections are installed there from the
existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
setup, vendored bytes are not source).
5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
losses on a consumer drive with NO power-loss protection, in a single failure domain.
etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
PLP is exactly how a member corrupts.
6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
one node is outweighed by the other two and causes node ejection. That holds for the
dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
faithfully to all three replicas, so quorum sees no disagreement — low probability, and
the only mitigation is different hardware. Follow-up noted: Longhorn's
snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
is the mechanism that operationalises the operator's own argument.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
337 lines
21 KiB
Markdown
337 lines
21 KiB
Markdown
# Prod bare-metal provisioning checklist (T-7.1)
|
||
|
||
**Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03
|
||
**Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod
|
||
hardware until the blockers in §7 are closed and `make -C ci gate` is green.
|
||
|
||
> **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6
|
||
> proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware
|
||
> ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes
|
||
> **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work
|
||
> knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry).
|
||
|
||
Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)).
|
||
The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to
|
||
call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse
|
||
the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation.
|
||
|
||
---
|
||
|
||
## 0. Node inventory
|
||
|
||
Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
|
||
|
||
| | node 1 | node 2 | node 3 |
|
||
|---|---|---|---|
|
||
| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
|
||
| LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
|
||
| Model | HP EliteDesk 800 G5 Desktop Mini | | |
|
||
| CPU | i7-9700T, 8c/8t, VT-x | | |
|
||
| RAM | 31 GiB | | |
|
||
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | |
|
||
| OS | Debian 13 (trixie), kernel 6.12 | | |
|
||
| Firmware | HP R21 Ver. 02.26.00 | | |
|
||
| `/dev/kvm` | present | | |
|
||
|
||
Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).
|
||
|
||
---
|
||
|
||
## 1. Firmware / BIOS (per node, before OS install)
|
||
|
||
- [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
|
||
boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
|
||
- [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
|
||
human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.)
|
||
- [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off).
|
||
- [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot
|
||
unless PXE is adopted.
|
||
- [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three
|
||
nodes match.
|
||
- [ ] **Firmware version recorded** in §0 and consistent across all three nodes.
|
||
|
||
## 2. Disk layout — **OPEN DECISION, blocks the build**
|
||
|
||
`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
|
||
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
|
||
(`docs/plans/ssd-etcd-isolation.md`).
|
||
|
||
Node 1 as-found is a **single** 477 GB SATA SSD carrying everything:
|
||
|
||
```
|
||
sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
|
||
├─sda1 976M /boot/efi
|
||
├─sda2 30G /
|
||
├─sda3 12G /var
|
||
├─sda4 977M swap
|
||
└─sda5 433G /srv <- the only large free space
|
||
```
|
||
|
||
Consequences to settle before imaging nodes 2 and 3:
|
||
|
||
- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
|
||
floor — see §5). Raw attributes, so the numbers are auditable:
|
||
|
||
| | |
|
||
|---|---|
|
||
| Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
|
||
| SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
|
||
| Power-on hours | 6,814 (~284 days) · 147 power cycles |
|
||
| **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
|
||
| **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
|
||
| **Erase failures** | **187** (`172`) |
|
||
| **Unexpected power losses** | **32** (`174`) |
|
||
| Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
|
||
| UDMA CRC errors | 0 (`199`) — cabling is clean |
|
||
|
||
**Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
|
||
count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
|
||
normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
|
||
not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
|
||
the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
|
||
uncorrectable counters instead.
|
||
|
||
**Verdict: usable, but not for production state.** Nothing says imminent failure — no
|
||
pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
|
||
consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
|
||
blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
|
||
power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
|
||
it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
|
||
a member corrupts — this drive has already logged 32 of them.
|
||
(Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
|
||
`DECISIONS-LOG.md`. It is not part of this verdict.)
|
||
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
|
||
(`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
|
||
Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
|
||
prod must override them in `group_vars`. On a single-disk node there is no second device at
|
||
all, so one of:
|
||
- **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
|
||
SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
|
||
spindle.
|
||
- **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
|
||
Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
|
||
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
|
||
physical failure domain.
|
||
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
|
||
- [ ] Record the chosen device paths here and set them in
|
||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
|
||
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
|
||
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
|
||
assert naming this section, instead of failing obscurely at `mkfs` time.
|
||
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
|
||
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
|
||
tasks this decision blocks — see §5's drive-health box for the exact command.
|
||
|
||
## 3. OS install (per node)
|
||
|
||
- [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline.
|
||
- [ ] Hostname `k3s-prod-N`.
|
||
- [ ] Partition per the §2 decision.
|
||
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
|
||
`roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
|
||
so the same role covers bare metal. Nothing to port.
|
||
|
||
## 4. Network (per node)
|
||
|
||
- [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway
|
||
`192.168.2.1`.
|
||
- [x] **Prod address plan — DECIDED (operator, 2026-08-31).**
|
||
|
||
| Role | Address | Owned by |
|
||
|---|---|---|
|
||
| `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory |
|
||
| **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) |
|
||
| **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) |
|
||
| **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) |
|
||
|
||
Two deliberate properties, both mirroring dev:
|
||
- The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and
|
||
Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way
|
||
round (`.49` control-plane, pool `.53-.60`).
|
||
- **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a
|
||
defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool
|
||
was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape
|
||
(`GATEWAY_VIP` `.53` = first of `.53-.60`).
|
||
- [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before
|
||
the build, and confirm nothing else on the network already answers on them.
|
||
- [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.**
|
||
Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single
|
||
upstream is a known SPOF that has already bitten this project — it broke containerd image
|
||
pulls and CoreDNS forwarding on dev.
|
||
|
||
The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
|
||
`roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
|
||
`/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
|
||
`/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
|
||
owns it), and configures the ordered upstream list either way. The earlier
|
||
`fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
|
||
drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
|
||
nodes, so it would have broken `node-baseline.yml` on every host.
|
||
|
||
What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
|
||
`node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
|
||
`BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
|
||
stops the play on it before writing anything. Choosing the second upstream is a **network
|
||
decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
|
||
resolve reliably but would also bypass any local zone and any filtering the gateway applies.
|
||
Decide between — and record the choice HERE, in the same change that edits the group_vars:
|
||
- a second on-LAN resolver on `192.168.2.x`;
|
||
- the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
|
||
prod sits on the workstation side of the pfSense boundary);
|
||
- an explicit, written-down acceptance of a public resolver as fallback-only.
|
||
|
||
Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
|
||
are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
|
||
servers, so keeping them means listing them explicitly.
|
||
`node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
|
||
local domain is decided.
|
||
- [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT —
|
||
prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind.
|
||
|
||
## 5. Access (per node) — **blocks all Ansible**
|
||
|
||
- [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1.
|
||
- [ ] **Password-protected sudo, driven by Ansible `become` + a vaulted password.**
|
||
**Prod does NOT use NOPASSWD sudo** (operator decision, 2026-08-31). Dev nodes do; prod
|
||
deliberately keeps the sudo password in place and supplies it to Ansible as an
|
||
**ansible-vault-encrypted** var, exactly like the existing `hypervisor` and `proxmox_hosts`
|
||
groups:
|
||
```
|
||
inventory/group_vars/k3s_prod/
|
||
README.md how to (re)create the vault — committed
|
||
vault.yml ansible_become_password: "..." — committed, ENCRYPTED
|
||
```
|
||
The passphrase that decrypts it lives **outside the repo** at
|
||
`~/.config/home-k3s/ansible-vault-pass` (`ansible.cfg :: vault_password_file`), so plays still
|
||
run unattended without a plaintext secret ever touching git.
|
||
```bash
|
||
ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ...
|
||
```
|
||
Prod nodes must therefore set `ansible_become: true` in their group_vars and must **not**
|
||
inherit any passwordless-sudo assumption from `group_vars/all.yml`.
|
||
- [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is
|
||
the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling,
|
||
2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part
|
||
of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline
|
||
would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead
|
||
(the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so
|
||
the placeholder device paths are never even templated):
|
||
```bash
|
||
cd infra/ansible
|
||
ansible-playbook playbooks/node-baseline.yml \
|
||
-e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
|
||
```
|
||
(Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading
|
||
the drive is then:
|
||
```bash
|
||
sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors
|
||
```
|
||
Record the result in §2 and close the layout decision in the same change that replaces the
|
||
two `node_baseline_*_device` placeholders.
|
||
|
||
## 6. Ansible baseline & cluster join
|
||
|
||
Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars
|
||
differ, plus a target-group override on the command line.
|
||
|
||
### How prod is targeted (done, 2026-08-31)
|
||
|
||
Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` /
|
||
`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately
|
||
kept separate because Ansible resolves them at different times:
|
||
|
||
| Layer | Knob | Set via |
|
||
|---|---|---|
|
||
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there |
|
||
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** |
|
||
|
||
So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and
|
||
tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with
|
||
no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after
|
||
`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**.
|
||
|
||
- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
|
||
**directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping.
|
||
(Nodes 2/3 stay commented out until they are physically on the bench.)
|
||
- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`),
|
||
`cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the
|
||
§2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which
|
||
are still BLOCKED placeholders.
|
||
- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until
|
||
it exists **no** prod play can escalate, and `--check` against prod cannot work either.
|
||
- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read:
|
||
```bash
|
||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
||
```
|
||
- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel
|
||
modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it
|
||
asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare
|
||
metal — the role already only *starts* it when the virtio channel exists, so it degrades
|
||
cleanly.)
|
||
- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`
|
||
→ HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three
|
||
nodes are present** — see §8.
|
||
- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the
|
||
prod LAN.
|
||
- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts
|
||
it afterwards. The bootstrap values and the HelmRelease **must stay in parity**
|
||
(`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config
|
||
(`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the
|
||
dev one the dev container exports as `$KUBECONFIG`.
|
||
- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`.
|
||
- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**:
|
||
`roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps
|
||
`--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a
|
||
target-group change. Tracked with the other repo blockers in §7.
|
||
- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015).
|
||
`playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=<group>`, but there is
|
||
deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted
|
||
PIA credentials — the existing vault is scoped to the dev group.
|
||
|
||
## 7. Repo blockers — **prod cannot be reconciled until these land**
|
||
|
||
`clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ
|
||
*only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests:
|
||
|
||
| Hardcoded value | Where | Prod needs |
|
||
|---|---|---|
|
||
| `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP |
|
||
| LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool |
|
||
| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
|
||
| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
|
||
| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
|
||
|
||
- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
|
||
overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
|
||
header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app
|
||
waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so
|
||
the seam exists.
|
||
- [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the
|
||
base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's
|
||
own comment flags this as owed "when hardware is built."
|
||
- [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is
|
||
byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`.
|
||
- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
|
||
the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
|
||
|
||
## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**)
|
||
|
||
Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench.
|
||
This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a
|
||
single-server-etcd window, and avoids a later member-add dance.
|
||
|
||
1. Per-node §1–§5 on each of the three boxes.
|
||
2. §7 repo work lands on `main` with the gate green.
|
||
3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
|
||
4. `flux bootstrap ... --path=clusters/prod`.
|
||
5. Conformance against prod (§9).
|
||
|
||
## 9. Exit criteria (Phase 7)
|
||
|
||
- [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02).
|
||
- [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
|
||
- [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod
|
||
specifically.
|
||
- [ ] Monitoring and alerts live on prod.
|
||
- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
|