jakman85andClaude Opus 5
666cc0d809
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the operator ruled for production. BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password` in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and proxmox_hosts groups. The decrypting passphrase stays outside the repo (~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext secret in git. Committed here: README.md + vault.yml.example only — the operator creates the encrypted vault (`ansible-vault create`), as with every other vaulted group. PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in node_baseline_packages, not hand-installed on a node. Drive health tooling is a permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it. Same click-ops-never principle as the rest of the repo. ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do not appear in Ansible. BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio names that do not exist on bare metal). Leaving them unset would silently fall back to those equally-nonexistent defaults — a confusing failure that looks like a wiring bug. This fails just as hard and says exactly why. KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not selected by them as-is. Harmless now (nothing can accidentally run against prod) but the "same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs. Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
116 lines
7.0 KiB
Markdown
116 lines
7.0 KiB
Markdown
# infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
|
|
|
|
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
|
|
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
|
|
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.
|
|
|
|
> **LIVE.** These plays have been executed against the running dev cluster (k3s
|
|
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
|
|
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.
|
|
|
|
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
|
|
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
|
|
|
|
## Prerequisites
|
|
1. `tofu apply` in `../tofu/` has created the VMs; cloud-init (`../cloud-init/`)
|
|
has baselined them (modules, sysctls, Longhorn disk, SSH user).
|
|
2. Inventory reflects the node IPs. `inventory/hosts.ini` ships aligned with the
|
|
LOCKED defaults; regenerate from tofu if overridden:
|
|
`cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
|
|
3. Run from the dev container (has `kubectl` + `helm`, used by the Cilium/verify/
|
|
Flux plays via `delegate_to: localhost`).
|
|
|
|
## Run order
|
|
```
|
|
cd infra/ansible
|
|
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
|
|
# — or step by step —
|
|
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
|
|
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
|
|
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
|
|
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
|
|
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
|
|
```
|
|
|
|
> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
|
|
> layers live OUTSIDE `site.yml` and must be run separately:
|
|
> ```
|
|
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); targets k3s_servers
|
|
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3
|
|
> ```
|
|
|
|
## Re-provisioning a node (tofu -replace / fresh OS)
|
|
A node rebuild wipes **everything on the host filesystem** — including host-level config
|
|
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
|
|
node-host layers or the node comes back missing pieces:
|
|
|
|
| Re-run | Why it doesn't survive a rebuild |
|
|
|---|---|
|
|
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
|
|
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
|
|
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
|
|
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |
|
|
|
|
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
|
|
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
|
|
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
|
|
should return a PIA exit IP (not the home ISP).
|
|
|
|
## What each layer does
|
|
| Playbook / role | Action |
|
|
|---|---|
|
|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
|
|
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
|
|
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
|
|
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |
|
|
|
|
## Cilium ownership / parity contract (docs/02)
|
|
Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
|
|
afterward. The parity-critical values **must match** between this role's
|
|
`templates/cilium-values.yaml.j2` and that HelmRelease:
|
|
- `kubeProxyReplacement: true`
|
|
- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
|
|
- `k8sServicePort: 6443`
|
|
|
|
The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
|
|
to the same value this role uses (surfaced by `tofu output primary_node_ip`).
|
|
|
|
## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
|
|
`roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
|
|
- `flux_bootstrap_sops_cmd` → `bootstrap/sops/install-age-key.sh` (Security
|
|
Engineer, T-1.4): installs the age **private** key as the `sops-age` Secret in
|
|
`flux-system`. Key never committed (`.gitignore` guards `age.key`/`*.key`).
|
|
- `flux_bootstrap_flux_cmd` → `bootstrap/flux/bootstrap.sh` (GitOps Engineer,
|
|
T-1.4): runs `flux bootstrap ... --path=clusters/dev`.
|
|
|
|
**These script names/args are assumptions** — if the bootstrap deliverables
|
|
expose different entrypoints, override the two vars in
|
|
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
|
|
for PM reconciliation.
|
|
|
|
## Prod inventory — scaffolding only, Phase 7 PREP (nothing above is run against it)
|
|
|
|
`inventory/hosts.ini` also carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
|
|
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
|
|
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
|
|
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
|
|
(vaulted `ansible_become_password` + non-secret overrides). Note `192.168.2.20` is prod's
|
|
**ingress** Gateway VIP (a Flux/`cluster-vars` concern, not an Ansible one — it does not appear in
|
|
`group_vars/k3s_prod/`); the kube-vip **control-plane** VIP (`kube_vip_address`) is a *separate*,
|
|
still-undecided address, left as a loud BLOCKED placeholder alongside the `node_baseline` device
|
|
overrides. See `group_vars/k3s_prod/README.md` and `infra/PROD-PROVISIONING.md` (T-7.1) §4/§2 for
|
|
the open decisions that still block a real run.
|
|
|
|
Everything else in this README describes the **dev** runbook. It does not (yet) apply to prod
|
|
as-is: the run order above and the roles it calls (`k3s_server`, `cluster_verify`, ...) target the
|
|
literal `k3s_primary` / `k3s_secondary` / `k3s_servers` groups, not `k3s_prod_*` — see the GAP note
|
|
at the top of `inventory/hosts.ini`'s prod section for what's still missing before that reuse (the
|
|
R-ENV-02 promise) actually works end to end.
|
|
|
|
## Static validation (PM, in dev container — NOT run here)
|
|
```
|
|
yamllint -c ../../ci/yamllint.yaml infra/ansible
|
|
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
|
|
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/
|
|
```
|