jakman85andClaude Opus 4.8
f2a7e42068
OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS +
40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn
prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway
API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a
cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate =
Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
71 lines
3.8 KiB
Markdown
71 lines
3.8 KiB
Markdown
# infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
|
|
|
|
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
|
|
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
|
|
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.
|
|
|
|
> **AUTHOR-ONLY — NOTHING HERE HAS BEEN EXECUTED.** No `ansible-playbook` run, no
|
|
> `tofu apply`, no cluster contacted. The PM static-validates (see below).
|
|
|
|
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
|
|
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
|
|
|
|
## Prerequisites
|
|
1. `tofu apply` in `../tofu/` has created the VMs; cloud-init (`../cloud-init/`)
|
|
has baselined them (modules, sysctls, Longhorn disk, SSH user).
|
|
2. Inventory reflects the node IPs. `inventory/hosts.ini` ships aligned with the
|
|
LOCKED defaults; regenerate from tofu if overridden:
|
|
`cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
|
|
3. Run from the dev container (has `kubectl` + `helm`, used by the Cilium/verify/
|
|
Flux plays via `delegate_to: localhost`).
|
|
|
|
## Run order
|
|
```
|
|
cd infra/ansible
|
|
ansible-playbook playbooks/site.yml # everything, in order
|
|
# — or step by step —
|
|
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
|
|
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
|
|
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
|
|
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
|
|
```
|
|
|
|
## What each layer does
|
|
| Playbook / role | Action |
|
|
|---|---|
|
|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
|
|
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
|
|
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
|
|
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |
|
|
|
|
## Cilium ownership / parity contract (docs/02)
|
|
Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
|
|
afterward. The parity-critical values **must match** between this role's
|
|
`templates/cilium-values.yaml.j2` and that HelmRelease:
|
|
- `kubeProxyReplacement: true`
|
|
- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
|
|
- `k8sServicePort: 6443`
|
|
|
|
The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
|
|
to the same value this role uses (surfaced by `tofu output primary_node_ip`).
|
|
|
|
## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
|
|
`roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
|
|
- `flux_bootstrap_sops_cmd` → `bootstrap/sops/install-age-key.sh` (Security
|
|
Engineer, T-1.4): installs the age **private** key as the `sops-age` Secret in
|
|
`flux-system`. Key never committed (`.gitignore` guards `age.key`/`*.key`).
|
|
- `flux_bootstrap_flux_cmd` → `bootstrap/flux/bootstrap.sh` (GitOps Engineer,
|
|
T-1.4): runs `flux bootstrap ... --path=clusters/dev`.
|
|
|
|
**These script names/args are assumptions** — if the bootstrap deliverables
|
|
expose different entrypoints, override the two vars in
|
|
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
|
|
for PM reconciliation.
|
|
|
|
## Static validation (PM, in dev container — NOT run here)
|
|
```
|
|
yamllint -c ../../ci/yamllint.yaml infra/ansible
|
|
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
|
|
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/
|
|
```
|