Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

docs: reconcile the documentation with the deployed reality

Browse Source 
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.

Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.

README.md
  - Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
  - Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
  - Real conformance number (36/51, 70.6%) and how to regenerate it.
  - Devcontainer workflow; note that the gate was red and why.

infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
  cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
  paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
  the toolchain, off-repo state that must not be lost, and the fact that Flux's
  git source is an app inside the cluster (`git push origin main` IS the deploy).

infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
  rationale are still worth keeping; the libvirt execution steps are not.

infra/README.md — Proxmox, plus a map of the subtree and where the cluster
  actually lives (it is not on this workstation).

infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
  is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
  talks to an HTTP API. Everything runs in the container now. Documents the full
  CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
  and the read-only spec repo especially — without the latter, spec-immutability
  silently skips instead of verifying).

infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
  is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
  against the running nodes), pfSense NAT marked done, and a warning that
  terraform.tfstate is absent on a fresh workstation — an apply with empty state
  would try to recreate the three live nodes.

infra/tofu/variables.tf — section comment disagreed with its own defaults.

infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
  domains that do not exist; behaviour unchanged, but it no longer lies.

apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
  since been dropped. Now: live status, 40Gi, resolved open questions marked
  resolved, and the two constraints the next person will otherwise rediscover —
  KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
  postBuild.substituteFrom (envsubst corrupts the cloud-init).

DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
  migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
  the Flux source cutover, CI going live in-cluster, ten app onboardings and the
  renderer bugs they exposed, the wave split (verified: DBs adopted, not
  recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
  VM, and kubectl SSO. Plus verified live state and the carried-forward open list.

Two corrections to the record while verifying:
  - kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
    [oidc:platform-admins]), so the rolling restart 3472f1d deferred has
    happened.
  - UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
    24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
    and the CNP explicitly permits cluster -> :11443, so it is genuinely not
    listening rather than a policy artifact.

Docs and comments only; no manifest semantics change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-08-27 00:00:05 -05:00
1 parent 4cf9572c58
commit 12c40d8c9c
10 files changed
+551 -147
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/PHASE1-SCOPE.md
+15 -1
View File
@@ -1,8 +1,22 @@
# Phase 1 Scope — dev/test Cluster Bring-up (local KVM)
> ## ⚠️ HISTORICAL — DO NOT EXECUTE
>
> **Phase 1 is complete.** This document is the *original* scoping for the first dev cluster, which
> ran on **local libvirt/KVM VMs on the workstation** (`10.73.0.0/24`). That cluster was retired: it
> was constrained by the workstation's ~22 GiB RAM and 137 GiB disk, and the dev cluster was migrated
> to a **Proxmox VE** host (nodes `192.168.1.50/.51/.52`, 6 vCPU / 16 GiB each).
>
> Kept for the record — the exit criteria, sizing rationale, and risk analysis below are still the
> reasoning behind the current design. **Everything about libvirt, `virsh`, the `dmacvicar/libvirt`
> provider, `10.73.0.x`, and the "missing prereqs" table is superseded.**
>
> For the cluster as it exists today, read **[`PHASE1-STATUS.md`](PHASE1-STATUS.md)**.
**Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md`
(Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux),
0007 (SOPS). **Status:** scoped, awaiting operator go / prerequisite installs.
0007 (SOPS). **Status (as written, 2026-08-16):** scoped, awaiting operator go / prerequisite
installs. **Status now:** delivered, then superseded by the Proxmox migration.
## Objective & exit criteria (roadmap Phase 1)
A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC
infra/PHASE1-STATUS.md
+120 -48
View File
@@ -1,57 +1,129 @@
# Phase 1 — Execution Status & Resume Guide
# Dev cluster — current state & reconnect guide
**Checkpoint: 2026-08-16.** The dev/test cluster is **built and running**. Phase 1 is paused before
the Flux/SOPS bootstrap (a deliberate decision — see "Remaining"). This doc is how a future session
picks up.
**Last verified: 2026-08-26.** Phase 1 is **complete and long since exited**; the cluster has been
running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
reconnects to it and re-establishes the toolchain.
## DONE — a live 3-node HA k3s + Cilium cluster (T-1.1/1.2/1.3)
- `tofu apply` created the `k3s-dev` NAT network (10.73.0.0/24) + 3 VMs (`k3s-dev-1/2/3`,
6 GiB/4 vCPU, 20 G OS + 40 G data, host-passthrough) — all **running**.
- cloud-init baselined the nodes; k3s HA (embedded etcd, 3 servers) + Cilium installed by Ansible.
- **Live health:** `kubectl get nodes` → **3/3 Ready**; `cilium status` → **OK**; CoreDNS up.
> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
> or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and
> [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc).
### Fixes applied during execution (committed to main)
- `ansible.cfg`: `stdout_callback yaml` removed in ansible-core 2.20 → builtin default.
- `cilium-values`: `egressGateway` requires `bpf.masquerade: true` (+ `enableIPv4Masquerade`) or the
agent fatals — added.
## The cluster
## Toolchain reality (important for resume)
The pinned **dev container** (`infra/devcontainer/`) holds flux/sops/age/kubectl/helm. BUT:
- **`tofu` runs on the HOST** — the `dmacvicar/libvirt` provider's read-write connect fails inside a
container (`Cannot find start time for pid`). The pinned binary was extracted to
**`~/.local/bin/tofu`** (OpenTofu 1.9.0). Run tofu from the host with `LIBVIRT_DEFAULT_URI=qemu:///system`.
- **`ansible` runs on the HOST** (pre-installed ansible-core 2.20; needs native network to the nodes).
## State locations (persisted on disk; gitignored)
| What | Path |
| | |
|---|---|
| tofu state | `infra/tofu/terraform.tfstate` |
| tofu vars (ssh pubkey) | `infra/tofu/terraform.tfvars` |
| cluster kubeconfig | `infra/ansible/.cache/dev.kubeconfig` (API at 10.73.0.11:6443) |
| SSH key | `~/.ssh/home_network` (user `k3s`, NOPASSWD sudo) |
| VMs / network | libvirt `qemu:///system` (`virsh list`, net `k3s-dev`) |
| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 16 GiB, Debian 12 (bookworm) |
| k3s | **v1.31.5+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
| CNI | Cilium 1.16.5 (kube-proxy replacement, L2 announcements, egress gateway) |
| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
| Storage | Longhorn 1.7.3 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |
Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium,
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`.
## Network path from the workstation (there is a NAT in the way)
The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two:
| Purpose | Workstation-side target | Forwards to |
|---|---|---|
| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` |
| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 |
| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host |
TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed.
## Access
**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates
against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the
kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`:
## Reconnect / manage
```bash
infra/cluster.sh status # VMs + kubectl get nodes
infra/cluster.sh start|stop # boot/graceful-shutdown the 3 VMs (state persists on disk)
eval "$(infra/cluster.sh kubeconfig)" # export KUBECONFIG
kubectl get nodes
ssh -i ~/.ssh/home_network [email protected]
```
The VMs are **running** at checkpoint (~18 GiB RAM). `infra/cluster.sh stop` frees RAM; k3s/etcd/
Cilium recover on `start` (data on the disks).
kubectl config use-context k3s-oidc
kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
```
## REMAINING Phase 1
- **T-1.4 Flux + SOPS bootstrap — Option A (decided):** start the host Gitea (`bootstrap/gitea/`),
push this repo, `flux install`, apply the `flux-system` GitRepository + age Secret — but **keep the
`platform` and `apps` Kustomizations SUSPENDED**. This proves Flux reconciles from the host Gitea
(Phase-1 exit) WITHOUT unleashing the incomplete Phase-0 platform stubs on the working cluster.
- **T-1.5** destroy/rebuild proof (`tofu destroy` a node → `apply` → re-join), then flip
`CLUSTER_AVAILABLE=true` so conformance L2/L3 runs.
`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so
this path needs an interactive desktop.
## Phase 2 prerequisite (before un-suspending `platform/`)
The Phase-0 `platform/` HelmReleases are stubs. **Do NOT let Flux reconcile them onto this cluster
until:** `platform/cilium` `k8sServiceHost` is set to **10.73.0.11** (parity with the Ansible install,
docs/02) and the REPLACE-ME values / secrets (Cloudflare token T-2.4p, etc.) are filled — otherwise
Flux would `helm upgrade` Cilium with a broken value and could break the running CNI.
**kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from
`/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and
add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the
gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need
this, since they cannot complete a browser login.
**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 [email protected]` (user `k3s`, NOPASSWD sudo).
Ports 2251/2252 for nodes 2/3.
## Toolchain — use the dev container, not the host
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate
toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
```bash
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
infra/devcontainer/dev.sh # interactive shell at /work
```
Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket,
which is what broke the old `dmacvicar/libvirt` provider in a container.
`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring —
**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase
file, and the sibling spec repo read-only at `/home-network-k3s` so the R-TEST-06 immutability
check runs instead of silently skipping.
> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`,
> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl`
> therefore needs an explicit `DEV_KUBECONFIG=<path to a break-glass cert kubeconfig>` — the OIDC
> context cannot work there anyway (no browser). Not yet fixed.
## Off-repo state you must not lose
| What | Where |
|---|---|
| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret |
| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` |
| Node SSH key | `~/.ssh/home_network` |
| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state |
| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes |
| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` |
## Health check
```bash
kubectl get nodes # 3/3 Ready
flux get kustomizations # flux-system, platform, platform-config,
# apps-wave-0/1/2, apps-vms — all Ready
kubectl get helmreleases -A # all Ready
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
kubectl get vmi -A # unifi Running
```
## Flux's git source is inside the cluster — mind the loop
`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**:
the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes
through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea,
the Gateway, CoreDNS and the wildcard cert all being up.
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap
Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that
is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept.
**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy.
There is no separate "apply" step and no GitHub remote.
infra/README.md
+28 -3
View File
@@ -1,4 +1,29 @@
# infra/ — Phase 1 node provisioning (local KVM)
# infra/ — node provisioning and the toolchain
OpenTofu (libvirt) + cloud-init + Ansible bring up the 3-node k3s/Cilium cluster.
See PHASE1-SCOPE.md. Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
OpenTofu (**Proxmox VE**) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: `../home-network-k3s/docs/02-environments-and-provisioning.md`.
| Path | What |
|---|---|
| [`tofu/`](tofu/) | The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (`bpg/proxmox` provider) |
| [`cloud-init/`](cloud-init/) | First-boot baseline shipped as a Proxmox user-data snippet |
| [`ansible/`](ansible/) | The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
| [`devcontainer/`](devcontainer/) | The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm **and the whole CI gate toolchain**) |
| [`cluster.sh`](cluster.sh) | **RETIRED** — drove the old libvirt VMs; does not work against Proxmox |
| [`PHASE1-STATUS.md`](PHASE1-STATUS.md) | **Current** state of the cluster and how to reconnect to it |
| [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) | Historical Phase-1 scoping doc (libvirt era — superseded) |
## Where the cluster actually lives
The dev cluster is **not** on this workstation. It runs on a Proxmox VE host; the nodes sit on the
Proxmox LAN and are reached across a pfSense NAT:
| | |
|---|---|
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52` — 6 vCPU, 16 GiB, Debian 12, k3s v1.31.5 |
| kube-API | kube-vip VIP `192.168.1.49:6443`, reached from the workstation via NAT `192.168.2.50:6443` |
| Node SSH | pfSense NAT `192.168.2.102:2250/2251/2252` → node `:22` (Ansible only) |
| Ingress VIP | `192.168.1.53` (Cilium L2, shared Gateway); `.53–.60` reserved for LB VIPs |
The earlier libvirt/KVM cluster on `10.73.0.0/24` is **retired**. Any doc or command referencing
`virsh`, `qemu:///system`, or `10.73.0.x` for the cluster is historical.
infra/cluster.sh
+13 -2
View File
@@ -1,6 +1,17 @@
#!/usr/bin/env bash
# infra/cluster.sh — manage the local dev k3s cluster (VMs + kubeconfig). Convenience for
# resuming Phase 1 development. Uses host virsh + kubectl (see infra/PHASE1-STATUS.md).
# infra/cluster.sh — RETIRED (libvirt era). DO NOT USE.
#
# This managed the ORIGINAL dev cluster: 3 libvirt/KVM VMs on the workstation, on the
# 10.73.0.0/24 NAT network. That cluster no longer exists — the dev cluster was migrated to
# Proxmox VE (nodes 192.168.1.50/.51/.52) and there are no local `k3s-dev-*` domains for
# virsh to start or stop. It is kept only as a record of the retired lifecycle.
#
# What to use instead (see infra/PHASE1-STATUS.md):
# VM power — the Proxmox UI / API on the Proxmox host, or `tofu` in infra/tofu
# kubeconfig — ~/.kube/config context `k3s-oidc` (Keycloak SSO), or the break-glass
# cert kubeconfig from the primary node
# node access — ssh -p 225{0,1,2} [email protected] (pfSense NAT)
# cluster health — kubectl get nodes && flux get kustomizations
set -euo pipefail
export LIBVIRT_DEFAULT_URI=qemu:///system
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
infra/devcontainer/README.md
+68 -36
View File
@@ -1,51 +1,83 @@
# Dev toolbox (containerized baseline)
The Phase 1 prerequisites are **not all Python** (OpenTofu, `age`, `sops`, `flux` are Go binaries;
only Ansible is Python). Per the operator's rule, the toolchain is therefore a **pinned dev
container** baked with everything needed to drive the local KVM cluster — no host pollution, one
reproducible baseline.
The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`,
`kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible
baseline, and the *same* binaries CI uses.
**This is the supported way to run anything in this repo.** Don't install the toolchain on the host.
## What's inside (pinned in `versions.env`)
OpenTofu, `age` + `age-keygen`, `sops`, `flux`, `kubectl`, `helm`, `ansible-core`, `libvirt-client`
(so the tofu `dmacvicar/libvirt` provider can talk to the host libvirt), `qemu-img`, cloud-init seed
tooling (`cloud-utils`/`genisoimage`), `openssh-clients`, and common utilities.
| Group | Tools |
|---|---|
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
| GitOps / cluster | `flux`, `kubectl`, `helm` |
| Secrets | `sops`, `age` + `age-keygen` |
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
The conformance-tool versions are kept in lockstep with `ci/versions.env` so
`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records
the resolved set at `/etc/dev-toolchain-versions`.
## Use
```bash
cd infra/devcontainer
./dev.sh build # build the image (needs network: base image + pinned binaries)
./dev.sh # build if needed + open a shell in /work (the repo)
./dev.sh run tofu -version
./dev.sh build # build the image (needs network)
./dev.sh # build if needed + shell in /work (the repo)
./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
./dev.sh run sops -d path/to.sops.yaml
./dev.sh run tofu -chdir=/work/infra/tofu plan
```
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
## How it reaches the host KVM
`dev.sh` runs the container with:
- `--network host` — so Ansible can SSH to the cluster VMs on the `k3s-dev` NAT subnet
(10.73.0.0/24) and `kubectl`/`flux` can reach the API server.
- `-v /run/libvirt:/run/libvirt` + `LIBVIRT_DEFAULT_URI=qemu:///system` — so OpenTofu/virsh drive
the host libvirt daemon (the socket is world-writable; no group change needed).
- `--security-opt label=disable` — lets the container use the host libvirt socket without
relabeling it (SELinux hosts).
- `-v $REPO:/work` and `-v ~/.ssh:/root/.ssh:ro` — the repo and your SSH keys.
## What `dev.sh` mounts, and why each one matters
## Known limitation — run `tofu` and `ansible` on the HOST, not in the container
Discovered during Phase 1 bring-up (see `../PHASE1-STATUS.md`):
- **OpenTofu + `dmacvicar/libvirt`** fails *inside* the container — its read-write `qemu:///system`
connect hits a polkit/PID-identity path (`Cannot find start time for pid`) that a mounted socket
can't satisfy across the PID namespace (`--pid=host` didn't help). **Run the pinned tofu on the
host** (extract once: `podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/`;
then `LIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...`).
- **Ansible** talks to the cluster VMs over SSH; simplest to run the pre-installed host `ansible-core`
with native networking to the `k3s-dev` NAT subnet.
- The container remains the baseline for **flux/sops/age/kubectl/helm** (used from inside with
`--network host` once the cluster exists).
- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
`kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API
(`192.168.2.102:8006`) on the host's network.
- `-v $REPO:/work` (workdir) — the repo.
- `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the
private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with
the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable.
- `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`).
- `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted
`group_vars` decrypt unattended.
- `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when
it exists beside this one. Without it `ci/spec-immutability.sh` (R-TEST-06) honestly *skips*
rather than passing — mount it, or that gate silently verifies nothing.
- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling.
## tofu and ansible now run **inside** the container
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a
container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and
run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which
talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs
SSH, which `--network host` provides.
So: run everything in here. The host needs only podman (or docker).
## kubectl inside the container — known gap
`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path
that no longer exists**, and does not mount `~/.kube`. Override it explicitly:
```bash
DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
```
Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser,
which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both
access paths. (Not yet fixed in `dev.sh`.)
## Notes
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
(one line) to the current stable and rebuild. The image records the resolved set at
`/etc/dev-toolchain-versions`.
- Secrets never enter the image. The SOPS **age private key** is generated at bootstrap and kept
out of git (`.gitignore` guards `age.key`).
- This container is the toolchain, not the cluster. It provisions the 3 KVM nodes (T-1.1–T-1.4);
the cluster itself runs on libvirt on the host.
and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`.
- **Secrets never enter the image** — they arrive only as the bind mounts listed above.
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md).
infra/tofu/README.md
+23 -10
View File
@@ -2,12 +2,15 @@
Provisions the "Create" lifecycle stage (docs/02): 3 cloud-init'd VMs on the
**Proxmox VE** box, each with an imported OS disk and a blank Longhorn data disk.
**Authoring only — nothing here has been applied.** (Migrated from the old
memory-constrained libvirt host; provider is now `bpg/proxmox`.)
(Migrated from the old memory-constrained libvirt host; provider is now `bpg/proxmox`.)
**APPLIED AND LIVE.** These three VMs are the running dev cluster — `k3s-dev-1/2/3`, k3s
v1.31.5, carrying the whole platform. Treat any `tofu apply/destroy` here as an operation on
production-for-dev, and note the state file caveat below.
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate).
## What it builds (defaults = VERIFIED Proxmox targets)
## What it builds (defaults = the LIVE cluster, verified 2026-08-26)
| Resource | Value |
|---|---|
@@ -16,11 +19,19 @@ Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate).
| Image/snippet datastore | `local` (dir; content = iso + **snippets**) |
| Bridge | `vmbr0` (no VLAN) |
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52`, gw `192.168.1.1`, DNS `192.168.1.18` |
| Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt) |
| OS disk | 40 GiB, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image |
| Data disk | 100 GiB, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init) |
| Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt — real KVM confirmed in-guest) |
| OS disk | **64 GiB**, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image |
| Data disk | **128 GiB**, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init; Longhorn reports ~125 GiB usable per node) |
| Cloud-init | native `initialization` (static IP + DNS) + user-data **snippet** for the baseline |
| Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs) |
| Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs). In use: `.53` shared Gateway, `.54` UniFi |
## ⚠️ State file
`terraform.tfstate` is gitignored and lives only on the workstation that ran the apply. On a
**fresh workstation it is absent** — and a `tofu apply` with empty state will try to *create* three
VMs that already exist. Recover the state file (or `tofu import` the existing VMs) before running
anything that plans changes. `tofu plan` against empty state is itself harmless, but read the plan:
"3 to add" means you are missing state, not that the cluster is gone.
All sizing/IPs/counts/placement are variables — see `variables.tf`.
@@ -45,9 +56,11 @@ All sizing/IPs/counts/placement are variables — see `variables.tf`.
without `https://`, the provider talks HTTP and eats a 301. Either fix the env
file to include the scheme, or export `TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT"`
so tofu normalizes it (prepends `https://`).
3. **pfSense SSH NAT** (network-dependent): the PM reaches these nodes and the k3s
API over a pfSense NAT that is not yet created; the Ansible inventory carries a
note to finalize `ansible_host` once the NAT rule exists.
3. **pfSense NAT** — **DONE, live.** The workstation reaches the nodes and the kube-API
across it: SSH `192.168.2.102:2250/2251/2252` → `192.168.1.50/.51/.52:22`, and kube-API
`192.168.2.50:6443` → kube-vip VIP `192.168.1.49:6443`. `infra/ansible/inventory/hosts.ini`
is **hand-maintained** for this mapping — do **not** regenerate it from
`tofu output -raw ansible_inventory`, which knows nothing about the NAT and would overwrite it.
## Auth (env only — never in tofu vars/state)
The provider reads these from the environment; the **API token is never a tofu
infra/tofu/variables.tf
+1 -1
View File
@@ -74,7 +74,7 @@ variable "node_count" {
}
}
# --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 40 GiB OS + 100 GiB data) --
# --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 64 GiB OS + 128 GiB data) --
variable "vcpu" {
description = "vCPUs (cores) per node. Generous — the box has plenty."