Phase 1 (author-only): node provisioning IaC — tofu + cloud-init + Ansible (T-1.1/1.2/1.3)

OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS + 40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate = Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 01:21:00 -05:00
1 parent 224cd8ac35
commit f2a7e42068
@@ -1,3 +1,45 @@
-# infra/cloud-init
+# infra/cloud-init — first-boot baseline templates

-_Phase 1 (T-1.x) — to be authored on execution go. See infra/PHASE1-SCOPE.md._
+Phase 1 **T-1.2**. The "Baseline" node lifecycle stage (docs/02 stage 2). These
+templates are consumed by the OpenTofu `libvirt_cloudinit_disk` in `../tofu/`
+(via `templatefile()`), one seed per node. **Authoring only — never applied.**
+
+Serves **R-ENV-01** (nodes are "created, baselined, cluster-joined, managed as
+code") and lays the kernel/OS floor ADR-0001 (k3s) + ADR-0002 (Cilium) +
+ADR-0003 (Longhorn) + ADR-0005 (KubeVirt) require.
+
+## Files
+- `user-data.yaml.tftpl` — `#cloud-config`: login user + injected SSH **public**
+  key, hostname, package baseline, kernel modules, sysctls, Longhorn data-disk
+  mount, swap-off.
+- `meta-data.yaml.tftpl` — NoCloud instance-id / local-hostname.
+- `network-config.yaml.tftpl` — netplan v2 **static** IP (DHCP is off on the NAT).
+
+`${...}` tokens are **OpenTofu** template vars filled by `../tofu/main.tf`
+(`hostname`, `fqdn`, `username`, `ssh_authorized_key`, `ip`, `prefix`, `gateway`)
+— they are not cloud-init/Jinja.
+
+## Baseline (cloud-init's job) vs. Ansible's job
+
+| cloud-init does (here) | Ansible does (T-1.3) |
+|---|---|
+| user + SSH public key, hostname | install k3s (HA, embedded etcd) |
+| static networking | join nodes, disable flannel/kube-proxy/traefik/servicelb |
+| `overlay`, `br_netfilter`, `iscsi_tcp`, `dm_crypt`, `nfs` modules | install Cilium (kube-proxy replacement, Gateway API, Hubble) |
+| bridge-nf + `ip_forward` sysctls, inotify limits | verify readiness |
+| open-iscsi + nfs-common (Longhorn client prereqs) | Flux + SOPS bootstrap handoff |
+| mount `/dev/vdb` at `/var/lib/longhorn` | — |
+| `swapoff` + strip swap from fstab | — |
+
+### Why these specifics
+- **overlay / br_netfilter + bridge-nf-call sysctls** — pod networking / iptables
+  bridging Cilium and kube-networking need.
+- **ip_forward / forwarding** — routing between pods, nodes, and egress.
+- **iscsi_tcp + open-iscsi (iscsid) and nfs / nfs-common** — Longhorn attaches
+  volumes over iSCSI and uses the NFS client for backup/RWX (ADR-0003).
+- **dm_crypt** — headroom for Longhorn volume encryption.
+- **swap off** — kubelet/k3s expectation.
+- **qemu-guest-agent** — lets libvirt/tofu read guest state.
+
+Nested-virt for KubeVirt (Phase 5) comes from the tofu `host-passthrough` CPU
+mode, not from here.
@@ -0,0 +1,3 @@
+# meta-data — NoCloud datasource identity. Rendered by OpenTofu templatefile().
+instance-id: ${hostname}
+local-hostname: ${hostname}
@@ -0,0 +1,19 @@
+# network-config (cloud-init Networking Config v2 / netplan). Rendered by OpenTofu.
+# Static addressing on the k3s-dev NAT: DHCP is off on the libvirt network, so each
+# node's IP comes from here. `match: name "en*"` avoids hard-coding ens3 vs enp1s0
+# across cloud images. Gateway == libvirt bridge host IP; it also serves DNS.
+version: 2
+ethernets:
+  primary:
+    match:
+      name: "en*"
+    dhcp4: false
+    dhcp6: false
+    addresses:
+      - ${ip}/${prefix}
+    routes:
+      - to: default
+        via: ${gateway}
+    nameservers:
+      addresses:
+        - ${gateway}
@@ -0,0 +1,75 @@
+#cloud-config
+# user-data — first-boot BASELINE only (docs/02 stage 2 "Baseline"). Rendered by
+# OpenTofu templatefile() ($${...} = tofu vars, NOT cloud-init). Everything here is
+# the node prerequisite floor for k3s / Cilium / Longhorn / KubeVirt. Cluster join,
+# k3s, and Cilium install are ANSIBLE's job (T-1.3) — deliberately NOT done here.
+
+hostname: ${hostname}
+fqdn: ${fqdn}
+preserve_hostname: false
+
+# Login user (also the Ansible ansible_user). Public key only — no passwords.
+users:
+  - name: ${username}
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: "ALL=(ALL) NOPASSWD:ALL"
+    lock_passwd: true
+    ssh_authorized_keys:
+      - ${ssh_authorized_key}
+
+# Baseline packages: guest agent (tofu/libvirt), and Longhorn's node prerequisites
+# (open-iscsi provides iscsid + iscsiadm; nfs-common provides the NFS client).
+package_update: true
+packages:
+  - qemu-guest-agent
+  - open-iscsi
+  - nfs-common
+  - curl
+
+# Kernel modules required by Cilium (overlay, br_netfilter) and Longhorn
+# (iscsi_tcp, dm_crypt, nfs). Loaded now and persisted for reboots.
+write_files:
+  - path: /etc/modules-load.d/k3s-cilium-longhorn.conf
+    permissions: "0644"
+    content: |
+      overlay
+      br_netfilter
+      iscsi_tcp
+      dm_crypt
+      nfs
+  - path: /etc/sysctl.d/99-k3s-cilium.conf
+    permissions: "0644"
+    content: |
+      net.bridge.bridge-nf-call-iptables  = 1
+      net.bridge.bridge-nf-call-ip6tables = 1
+      net.ipv4.ip_forward                 = 1
+      net.ipv6.conf.all.forwarding        = 1
+      fs.inotify.max_user_instances       = 8192
+      fs.inotify.max_user_watches         = 524288
+
+# Dedicated Longhorn data disk (/dev/vdb) → ext4 mounted at /var/lib/longhorn.
+# nofail so a node still boots if the disk is absent during a rebuild.
+fs_setup:
+  - label: longhorn
+    filesystem: ext4
+    device: /dev/vdb
+    partition: none
+    overwrite: false
+mounts:
+  - ["LABEL=longhorn", "/var/lib/longhorn", "ext4", "defaults,nofail,x-systemd.device-timeout=5s", "0", "2"]
+
+runcmd:
+  # Apply modules + sysctls now (cloud-init runs after they're written).
+  - ["modprobe", "overlay"]
+  - ["modprobe", "br_netfilter"]
+  - ["modprobe", "iscsi_tcp"]
+  - ["sysctl", "--system"]
+  # Longhorn needs iscsid running; guest agent for libvirt/tofu.
+  - ["systemctl", "enable", "--now", "iscsid"]
+  - ["systemctl", "enable", "--now", "qemu-guest-agent"]
+  # k3s/kubelet baseline: disable swap.
+  - ["swapoff", "-a"]
+  - ["sed", "-i", "/\\sswap\\s/d", "/etc/fstab"]
+
+# NOTE: no reboot here — the baseline is live after runcmd. Ansible connects next.