Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Merge docs/reconcile-with-reality: docs match reality; merge gate green again

Browse Source 
Reconciles the documentation with the deployed platform (the root README still
claimed Phase 0 / no cluster; the infra docs described a retired libvirt cluster;
DECISIONS-LOG.md had gone ~85 commits stale), and fixes the two independent
defects that had left the merge gate BLOCKED on main:

  - yamllint commas errors in the hand-authored apps/vms/unifi manifests
    (whitespace only; kustomize output byte-identical);
  - ci/touched-requirements.sh scanning whole file contents instead of the diff,
    which mis-attributed requirements to changes that never touched them and made
    appending to the decision log fail CI.

Gate verified in the devcontainer: spec-immutability / lint / l0 / l1 green,
MERGE GATE: PASS.
This commit is contained in:
jakman85 committed 2026-08-27 00:12:02 -05:00
2 parents 3472f1d161 9e8b694ba9
commit 91c82d2756
17 files changed
+665 -183
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

DECISIONS-LOG.md
+181
View File
@@ -922,3 +922,184 @@ R-TLS-01/03 GREEN** (rewrote the probe for LE-prod system-trust + split-horizon
LESSON: heavy sustained node load (TCG-emulated VMs) can degrade Cilium datapath -> intermittent LB-VIP LESSON: heavy sustained node load (TCG-emulated VMs) can degrade Cilium datapath -> intermittent LB-VIP
reachability; a restart clears it. NOTE: after a host reboot, restart the bootstrap Gitea container reachability; a restart clears it. NOTE: after a host reboot, restart the bootstrap Gitea container
(`bash bootstrap/gitea/bootstrap-gitea.sh`) - it is Flux's source and does not auto-start; the k3s VMs auto-recover. (`bash bootstrap/gitea/bootstrap-gitea.sh`) - it is Flux's source and does not auto-start; the k3s VMs auto-recover.
### BACKFILL — Proxmox migration, app tier, wave split, UniFi VM, kubectl SSO (2026-08-25 → 08-26)
**Process note first:** this log had gone stale at `1eb2cb7` (2026-08-25 AM) while `main` advanced
**~85 commits** to `3472f1d`. Everything below is reconstructed from those commits and **verified
against the live cluster on 2026-08-26**. The lesson is the obvious one: the log is the PM's
authoritative record and a two-day gap in it cost a session's worth of re-derivation. Append per
work item, not per session.
**1. INFRA — the dev cluster MIGRATED off libvirt onto Proxmox VE.** The workstation-hosted libvirt
cluster (3 VMs on `10.73.0.0/24`, 6 GiB each) was RAM/disk-starved — the KubeVirt work was OOMing
nodes (`971c730` had to temp-disable KubeVirt just to relieve pressure). Provisioning was ported
`dmacvicar/libvirt` → **`bpg/proxmox`** (`53ca698`), cloud-init moved from a netplan/meta-data seed
ISO to a Proxmox **user-data snippet** and then to API-native cloud-init with the rich baseline
pushed into the Ansible `node_baseline` role (`39123d7`, `a2392a4`). Nodes are now
**`192.168.1.50/.51/.52`, 6 vCPU / 16 GiB / 64 GiB OS + 128 GiB Longhorn data** (`8a8f28b`), Debian
12, `cpu_type=host` for nested KVM. Gotchas paid for along the way: the base image must be
downloaded as **import** content-type for `import_from` (`fca8d8f`); the Debian *genericcloud*
image kernel-panics on first boot under bpg — use the **generic** image + serial console
(`28d200c`, upstream bpg#1639); `qemu-guest-agent` must be enable-only when tofu leaves the virtio
channel off, or the baseline play fails (`1004c8e`).
**2. NETWORK — pfSense NAT, then kube-vip, then the tunnel died.** The workstation is not on the
cluster LAN. First cut routed Ansible over a pfSense SSH NAT (`192.168.2.102:225x` → node `:22`)
and decoupled `k3s_node_ip` from `ansible_host` (`1ff2531`) — `inventory/hosts.ini` is therefore
**hand-maintained and must never be regenerated** from `tofu output ansible_inventory`. The
kube-API initially needed an SSH tunnel (`0037fb4`). That was retired by **kube-vip**: a
control-plane HA VIP at **`192.168.1.49`**, added to the k3s `tls-san` list so TLS validates
through the NAT via `tls-server-name` (`60a58f0`, `81ef782`). Controller access is now direct:
`https://192.168.2.50:6443` → VIP. Gateway/ingress VIP moved to **`192.168.1.53`** on the Proxmox
LAN, and `GATEWAY_VIP` + the split-horizon CoreDNS answer + the DNS conformance tests were all
repointed (`604552e`, `63b87ff`).
**3. COLD-START DEADLOCK, again — and the durable lesson.** Bringing the platform up on the new
cluster re-hit the Phase-6 shape: `oauth2-proxy` sits in the `platform` `wait:true` tier but needs
Keycloak + trusted TLS + split-horizon DNS, so while it Failed the whole tier stayed not-Ready and
`platform-config` (which applies the fix) could never reconcile. Broken the same way as before —
temp-disable `oauth2-proxy` and its orphaned `platform-config` refs (`2fea1e2`, `efd6dbe`), let
platform go Ready, then re-enable (`e6b3a03`). KubeVirt/CDI re-enabled once the 16 GiB nodes
existed (`21669da`). **The FOLLOW-UP from Session 7 is still open and is now twice-proven: move
`oauth2-proxy` to a post-platform tier (`dependsOn: platform-config`) so a leaf forward-auth
service cannot gate its own dependencies.**
**4. FLUX SOURCE — now the in-cluster Gitea.** Bootstrap source repointed (`fff4499`), Gitea fixed
to mount its PVC at `/data` with a headless-install bootstrap (`73c4475`), then
`GitRepository/flux-system` cut over to **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**
(`d99d171`). Note the loop this creates: Flux's source is an app Flux deploys, reached via
split-horizon DNS → Gateway VIP. Stable in practice (Gitea's Deployment/PVC survive reboots
without Flux), but if Gitea is ever *lost* rather than restarted, recovery is the host bootstrap
Gitea in `bootstrap/`. `git push origin main` **is** the deploy — there is no other apply step.
**5. CI — the merge gate is now REAL and in-cluster.** Gitea Actions enabled with an in-cluster
`act_runner` + DinD, running the static gate as the single required check on `main` (`bc4519b`).
Fixes: kyverno CLI arch is `x86_64` not `amd64`, and the job must wait for DinD certs (`79e7f92`);
`make` masked spec-immutability's exit 3 as a 2, so `gate.sh` calls the script directly
(`0f75fa2`); `upload-artifact@v4` is unsupported by Gitea Actions (`67bc317`). The **whole test
toolchain is now baked into the devcontainer** (`bb48f67`) so `make -C ci gate` on a laptop is
byte-identical to CI (R-TEST-02). Kyverno itself needed two real fixes: the `default-deny-shape`
policy's scalar-array pattern was **inverted** and silently accepted Ingress-only policies — now
expressed via `deny.conditions` requiring both (`1b0848c`); and the fail-closed admission
controller was crashlooping into 502s until its resources were raised and dev replicas set to 2
(`28a34c4`). Chainsaw parallelism dropped 4→2, then heavy L2/L3 tests serialized, to stop dev etcd
overload cascades (`9f52f19`, `948c390`).
**6. APP TIER — 10 services onboarded through the renderer.** gitea, plex, sonarr, radarr,
qbittorrent, jackett (`b76fb7b`…`8392b52`), then seerr (`b6f72ea`), nextcloud (`4be6752`) and
semaphore (`77496f9`) — the last two being the **first CNPG-DB + OIDC-target patterns**. Renderer
bugs found by real onboarding, each fixed in the renderer rather than by hand-patching output:
world/internet egress allow emitted per egress-class (`7e8ae47`); **reciprocal ingress-allow** for
declared dependencies, without which a declared dep silently did not connect under default-deny
(`14beb85`, R-NET-02); east-west selectors **namespace-pinned** so they match the right pods
(`b871877`); `Recreate` strategy for Deployments mounting RWO PVCs, else rollouts deadlock
(`d32b5c4`). ServiceDefinition schema gained `env`/`envFrom`/`lifecycle`, retiring the bespoke
gitea bootstrap patch (`c8dd4f3`). Semaphore cost two root-causes: `fsGroup` via schema-1.3.0
`podSecurityContext` for non-root PVC write (`28498f1`), and **`SEMAPHORE_PORT` being shadowed by
kubelet service-link env injection** (`41d696e`) — a trap for any app whose config env name
collides with a Service name.
**7. APP TIER TOPOLOGY — monolithic `apps` split into WAVES.** One `wait:true` Kustomization over
all ~130 app resources meant one crashloop blocked every app and every new revision (it bit twice).
Split into `apps-wave-0/1/2`, each `dependsOn` the previous, wave assignment computed by
`catalog/render.py::compute_waves()` from the declared dependency graph. Design + the data-loss
cutover analysis are in **`DESIGN-waves.md`**. The cutover was executed as designed: `prune:false`
on the old `apps` first (`1bfb0db`), then the split (`36a1a4e`) so the waves **adopt** by identity
instead of delete/recreate. **VERIFIED 2026-08-26:** `nextcloud-db` carries
`kustomize.toolkit.fluxcd.io/name=apps-wave-0`, the CNPG Clusters retain their ORIGINAL ages
(22h/21h — not recreated), and `prune:true` is restored on the waves. No data was lost.
**8. PHASE 5 — storage for real VMs, and the UniFi VM.** CSI snapshots + smart-clone + a migratable
RWX `longhorn-vm` StorageClass landed to make KubeVirt **live migration** possible (`3586769`);
two KubeVirt alerts were plain wrong and firing falsely — a broken http CDI ServiceMonitor causing
`CDIOperatorDown` (`6a7e9d1`) and `KubeVirtControlPlaneDown` using a non-existent metric with the
wrong job label (`b57b3e9`). **UniFi OS Server** was then onboarded as a VM on a deliberately
**non-gating** track (`317b3f0`): `apps/vms/` reconciled by `apps-vms` (`wait:false`,
`dependsOn: platform-config`), because a VM's image import + in-guest install is legitimately
long-not-Ready and must never gate the app tier. Four cluster-level truths came out of it:
- **`nonroot-devices: true`** on the k3s servers — the non-root CDI importer must open a raw Block
device k8s never chowns for it (`df5979f`).
- **`longhorn-scratch`**, a single-replica class for CDI scratch; the 3-replica default failed the
import with insufficient storage (`7a3da96`).
- **KubeVirt caps inline `userData` at 2048 bytes.** The rich cloud-init exceeded it; a
`userDataSecretRef` was tried and reverted; shipped state is a **minimal ~1.1 KB inline**
payload (`e99e3d1`, `760084e`, `ab59202`). The `install-uos` helper did not survive the cut.
- **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux envsubst rewrote `${...}` inside
the cloud-init and corrupted it (`da1f833`). Do not restore it for consistency with the waves.
**9. PHASE 6 — kubectl login via Keycloak SSO (`3472f1d`).** A **public** `kubernetes` client with
PKCE (no secret in any kubeconfig), a groups mapper, kube-apiserver OIDC flags via the k3s_server
role, and `k8s-oidc-rbac.yaml` binding `oidc:platform-admins` → `cluster-admin`,
`oidc:viewers` → `view`. The cert kubeconfig is untouched and remains a fully independent
**break-glass** path. **VERIFIED LIVE:** `kubectl auth whoami` returns
`oidc:<user>` with `[oidc:platform-admins]` — so the API-server flags are active; the rolling
restart the commit message deferred has since happened.
**LIVE STATE VERIFIED 2026-08-26 (fully green):** 3/3 nodes Ready (k3s v1.31.5); **7/7 Flux
Kustomizations Ready** at `3472f1d` (flux-system, platform, platform-config, apps-wave-0/1/2,
apps-vms); **10/10 HelmReleases Ready**; zero failing pods; 3 CNPG Clusters healthy (keycloak,
nextcloud, semaphore); UniFi VMI Running with guest agent connected (Ubuntu 24.04.4,
LiveMigratable). Conformance scaffold coverage **36/51 (70.6%)**.
**GATE WAS RED — two independent defects found and FIXED (branch `docs/reconcile-with-reality`).**
`main` had been failing the merge gate since the UniFi commits landed. Two separate causes:
1. **yamllint `commas` errors** in the hand-authored `apps/vms/unifi/` manifests
(`30-virtualmachine.yaml`, `40-networkpolicy.yaml`, `50-service-l4.yaml` — column-aligned inline
flow mappings). Cosmetic, but `lint` blocks the merge, so the VM work went in without a passing
gate. Fixed by normalizing the spacing; `kubectl kustomize apps/vms` renders **byte-identical**
before and after. The lint config was deliberately NOT relaxed — loosening a rule to make failing
files pass is moving the goalposts.
2. **`ci/touched-requirements.sh` mis-attributed requirements** — the real defect, and the one
previously logged as "FOLLOW-UP (WS-TEST): fix the matcher so real requirements need not be
exempted". The script's own header says it scans "R-* tokens literally present in the **diff**",
but it actually grepped **whole file contents**. So any edit to a long-lived file pulled in every
requirement that file had ever mentioned in prose: appending one entry to **this log** dragged in
`R-SEC-01`/`R-SEC-03`/`R-TEST-06`, and touching `infra/tofu/` dragged in `R-ENV-01`/`R-PLAT-01`.
Several are legitimately uncovered, so `--require-all` then **BLOCKED the merge on lines the
author never touched** — i.e. the PM could not append to the decision log without failing CI.
Fixed with two scoping rules: scan the added/removed lines of the diff (as the header always
claimed), and **skip `*.md`** — an ID in prose is a *reference*, not a declaration. Without the
second rule the log cannot record which requirements a change concerned without thereby claiming
to have touched them, and the very commit that explains a mis-attribution re-triggers it.
Removed lines still count (deleting a `conformance.home.k3s/R-*` tag touches that requirement too).
**Known consequence, accepted:** a manifest whose *header* declares the requirements it serves
(e.g. `platform/keycloak/keycloak-realm-import.yaml`) no longer contributes those IDs unless the
change touches a line carrying them. That is the intended semantics — the harness debt was
written as "stop keying on **doc-header tokens**" — and the authoritative signal, a test's
`conformance.home.k3s/R-*` label, is still detected. Verified: a range spanning conformance and
platform YAML changes still resolves R-NET-01, R-OBS-02/04, R-STATE-01/02 and R-TEST-01..04.
**This is a gate-semantics change — flagged for review.** It is a tightening of attribution, not
a weakening: no requirement is exempted, `DOCUMENTED_GAPS` is untouched, authoritative
declarations (test tags, Kyverno policies, ServiceDefinitions) all live in YAML on lines a real
change necessarily touches, and the doc subtrees that matter (`conformance/`, `schemas/`,
`catalog/`, `clusters/`, `ci/`) stay covered by the path→requirement fallback regardless. The alternative fixes were worse:
adding five more `DOCUMENTED_GAPS` genuinely weakens the gate, and scrubbing requirement IDs out
of prose (precedent `f9e9640`) would mean vandalizing this log's own history to appease a matcher.
Gate after both fixes, run in the devcontainer: spec-immutability ✅ (spec pinned at `b633b7f`,
matching the sibling repo), lint ✅, Kyverno L0 ✅, L1 render 3/3 ✅ — **MERGE GATE: PASS**.
**OPEN / NEXT (carried forward):**
1. **Install UniFi OS Server in the guest** — the VM runs bare Ubuntu; `:11443` answers nothing
(CNP permits `cluster`→`:11443`, so it is genuinely not listening). Install is interactive with
a rotating signed URL — needs an operator decision (hand-install vs. a verified silent path).
2. **`R-STATE-02`** — the last safety-critical requirement with no L3 test (Longhorn node-loss).
Currently exempted via `DOCUMENTED_GAPS`.
3. **oauth2-proxy → post-platform tier** (twice-proven cold-start deadlock).
4. **OQ-BKP** — off-cluster backup target; gates Velero, CNPG DR depth, and `R-STATE-03`.
5. **OQ-2** — the `bulk` StorageClass exists but has `server/share: REPLACE-ME` and provisions
nothing; plex/arr/qbittorrent ship config-only until it is resolved.
6. **image-automation** still authored-but-disabled (needs a Gitea write-cred).
7. **Coverage matcher, remaining half.** The content-scan half is fixed (see above). The coarse
`path_requirements` half still over-matches: any `clusters/*` edit claims `R-ENV-02` +
`R-GITOPS-01`, which is why both remain in `DOCUMENTED_GAPS`. Tighten the path map, or scaffold
those two, and drop the exemptions (WS-TEST).
8. **`dev.sh` KUBECONFIG** defaults to the dead libvirt-era path
`infra/ansible/.cache/dev.kubeconfig` and `~/.kube` is not mounted — in-container kubectl needs
an explicit `DEV_KUBECONFIG` pointing at a break-glass cert kubeconfig (OIDC needs a browser).
9. **tofu state** is absent on a fresh workstation; a `tofu apply` with empty state would try to
recreate the three live nodes. Recover the state file or `tofu import` before planning changes.
10. **Docs hygiene.** This log going two days stale is what made the whole reconciliation necessary.
Append per work item, not per session.
README.md
+57 -11
View File
@@ -16,16 +16,62 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
## Layout ## Layout
``` ```
platform/ platform services (Flux HelmReleases/Kustomizations): cilium, longhorn, infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
cert-manager, externaldns, gateway, monitoring, velero, kubevirt, pinned dev container that carries the whole toolchain
image-automation, keycloak, oauth2-proxy, kyverno/ bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
apps/ application workloads, one dir per catalog service platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
catalog/ ServiceDefinition entries (the onboarding surface) longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
schemas/ ServiceDefinition JSON Schema + renderer contract vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
clusters/ Flux entrypoints + dev/ and prod/ overlays platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
conformance/ Chainsaw behavior/conformance suite + harness + coverage report (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
.gitea/ Gitea Actions CI (lint · schema · Kyverno L0 · Chainsaw trigger · merge gate) Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
``` ```
Current phase: **Phase 0 — Foundations** (repo, CI, ServiceDefinition schema, conformance harness, ## Status
Kyverno L0 gate). No cluster exists yet; Phase 1 (cluster bring-up) is not authorized.
Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
open** — `clusters/prod` is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
the full report is `conformance/_artifacts/coverage/coverage.md`.
The running narrative — decisions, root causes, incidents, follow-ups — is
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
## Working on this repo
Everything the gate needs is baked into the pinned dev container — **do not install the
toolchain on the host**:
```bash
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
```
The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
[`infra/devcontainer/README.md`](infra/devcontainer/README.md).
A green run prints `MERGE GATE: PASS`. The L2/L3 conformance layers stay `SKIPPED` unless
`CLUSTER_AVAILABLE=true` — the static gate never pretends to have run them.
apps/vms/unifi/10-default-deny.yaml
+1 -1
View File
@@ -1,5 +1,5 @@
--- ---
# STAGING / DRAFT — see namespace.yaml header. # LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header).
# #
# Default-deny baseline (R-NET-01): deny all ingress + egress, allow ONLY DNS to the # Default-deny baseline (R-NET-01): deny all ingress + egress, allow ONLY DNS to the
# in-cluster resolver. Identical shape to the rendered pihole/app baseline. The VM's # in-cluster resolver. Identical shape to the rendered pihole/app baseline. The VM's
apps/vms/unifi/20-datavolume.yaml
+2 -2
View File
@@ -1,5 +1,5 @@
--- ---
# STAGING / DRAFT — see namespace.yaml header. # LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header).
# #
# CDI DataVolume — the UniFi VM boot disk. This is the FIRST real CDI *http* import on # CDI DataVolume — the UniFi VM boot disk. This is the FIRST real CDI *http* import on
# this cluster (the vm-demo/conformance path imports a containerdisk from a *registry*; # this cluster (the vm-demo/conformance path imports a containerdisk from a *registry*;
@@ -9,7 +9,7 @@
# STANDALONE DataVolume (NOT a dataVolumeTemplate) so the disk OUTLIVES `kubectl delete vm` # STANDALONE DataVolume (NOT a dataVolumeTemplate) so the disk OUTLIVES `kubectl delete vm`
# and survives destroy/rebuild (R-STATE-01) — same durability contract as vm-demo. # and survives destroy/rebuild (R-STATE-01) — same durability contract as vm-demo.
# #
# STORAGE = longhorn-vm (the RWX/migratable StorageClass the storage workstream is adding): # STORAGE = longhorn-vm (the RWX/Block/migratable StorageClass; live in platform/longhorn):
# - accessModes: ReadWriteMany + volumeMode: Block => REQUIRED for KubeVirt LIVE MIGRATION # - accessModes: ReadWriteMany + volumeMode: Block => REQUIRED for KubeVirt LIVE MIGRATION
# (evictionStrategy: LiveMigrate in the VM). A ReadWriteOnce disk cannot live-migrate. # (evictionStrategy: LiveMigrate in the VM). A ReadWriteOnce disk cannot live-migrate.
# - 40Gi: the Ubuntu cloud image virtual size is ~3.5Gi; cloud-init growpart/growroot # - 40Gi: the Ubuntu cloud image virtual size is ~3.5Gi; cloud-init growpart/growroot
apps/vms/unifi/30-virtualmachine.yaml
+8 -8
View File
@@ -65,14 +65,14 @@ spec:
# these and (b) surfaces them as containerPorts on the virt-launcher pod so the # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
# LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them. # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
ports: ports:
- { name: inform, port: 8080, protocol: TCP } # device inform / adoption - { name: inform, port: 8080, protocol: TCP } # device inform / adoption
- { name: ui-legacy, port: 8443, protocol: TCP } # legacy Network UI (still served) - { name: ui-legacy, port: 8443, protocol: TCP } # legacy Network UI (still served)
- { name: ui, port: 11443, protocol: TCP } # UOS web UI (modern default) - { name: ui, port: 11443, protocol: TCP } # UOS web UI (modern default)
- { name: portal-http, port: 8880, protocol: TCP } # guest portal HTTP - { name: portal-http, port: 8880, protocol: TCP } # guest portal HTTP
- { name: portal-https, port: 8843, protocol: TCP } # guest portal HTTPS - { name: portal-https, port: 8843, protocol: TCP } # guest portal HTTPS
- { name: throughput, port: 6789, protocol: TCP } # mobile speedtest / throughput - { name: throughput, port: 6789, protocol: TCP } # mobile speedtest / throughput
- { name: stun, port: 3478, protocol: UDP } # STUN (device<->controller) - { name: stun, port: 3478, protocol: UDP } # STUN (device<->controller)
- { name: discovery, port: 10001, protocol: UDP } # L2 device discovery - { name: discovery, port: 10001, protocol: UDP } # L2 device discovery
resources: resources:
requests: requests:
memory: 4Gi memory: 4Gi
apps/vms/unifi/40-networkpolicy.yaml
+7 -7
View File
@@ -1,5 +1,5 @@
--- ---
# STAGING / DRAFT — see namespace.yaml header. # LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header).
# #
# Additive ALLOWs on top of the 10-default-deny baseline. Cilium enforces the UNION of the k8s # Additive ALLOWs on top of the 10-default-deny baseline. Cilium enforces the UNION of the k8s
# NetworkPolicy (default-deny) and these CiliumNetworkPolicies (same pattern as pihole). The VM's # NetworkPolicy (default-deny) and these CiliumNetworkPolicies (same pattern as pihole). The VM's
@@ -31,13 +31,13 @@ spec:
- host - host
toPorts: toPorts:
- ports: - ports:
- { port: "8080", protocol: TCP } # inform / adoption - { port: "8080", protocol: TCP } # inform / adoption
- { port: "11443", protocol: TCP } # UOS UI - { port: "11443", protocol: TCP } # UOS UI
- { port: "8443", protocol: TCP } # legacy UI - { port: "8443", protocol: TCP } # legacy UI
- { port: "8880", protocol: TCP } # guest portal http - { port: "8880", protocol: TCP } # guest portal http
- { port: "8843", protocol: TCP } # guest portal https - { port: "8843", protocol: TCP } # guest portal https
- { port: "6789", protocol: TCP } # throughput - { port: "6789", protocol: TCP } # throughput
- { port: "3478", protocol: UDP } # STUN - { port: "3478", protocol: UDP } # STUN
- { port: "10001", protocol: UDP } # discovery - { port: "10001", protocol: UDP } # discovery
--- ---
# 2) EGRESS from the controller: internet (apt, the UOS installer download, UniFi updates + remote # 2) EGRESS from the controller: internet (apt, the UOS installer download, UniFi updates + remote
apps/vms/unifi/50-service-l4.yaml
+8 -8
View File
@@ -1,5 +1,5 @@
--- ---
# STAGING / DRAFT — see namespace.yaml header. # LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header).
# #
# ============================================================================================ # ============================================================================================
# L4 EXPOSURE (the hard part) — why the shared HTTPS Gateway is NOT enough for UniFi. # L4 EXPOSURE (the hard part) — why the shared HTTPS Gateway is NOT enough for UniFi.
@@ -49,12 +49,12 @@ spec:
selector: selector:
kubevirt.io/vm: unifi kubevirt.io/vm: unifi
ports: ports:
- { name: inform, port: 8080, targetPort: 8080, protocol: TCP } - { name: inform, port: 8080, targetPort: 8080, protocol: TCP }
- { name: ui, port: 11443, targetPort: 11443, protocol: TCP } - { name: ui, port: 11443, targetPort: 11443, protocol: TCP }
- { name: ui-legacy, port: 8443, targetPort: 8443, protocol: TCP } - { name: ui-legacy, port: 8443, targetPort: 8443, protocol: TCP }
- { name: portal-http, port: 8880, targetPort: 8880, protocol: TCP } - { name: portal-http, port: 8880, targetPort: 8880, protocol: TCP }
- { name: portal-https, port: 8843, targetPort: 8843, protocol: TCP } - { name: portal-https, port: 8843, targetPort: 8843, protocol: TCP }
- { name: throughput, port: 6789, targetPort: 6789, protocol: TCP } - { name: throughput, port: 6789, targetPort: 6789, protocol: TCP }
--- ---
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
@@ -72,5 +72,5 @@ spec:
selector: selector:
kubevirt.io/vm: unifi kubevirt.io/vm: unifi
ports: ports:
- { name: stun, port: 3478, targetPort: 3478, protocol: UDP } - { name: stun, port: 3478, targetPort: 3478, protocol: UDP }
- { name: discovery, port: 10001, targetPort: 10001, protocol: UDP } - { name: discovery, port: 10001, targetPort: 10001, protocol: UDP }
apps/vms/unifi/README.md
+79 -35
View File
@@ -1,8 +1,18 @@
# UniFi Network controller (UniFi OS Server) — VM onboarding **[STAGING / DRAFT]** # UniFi Network controller (UniFi OS Server) — VM onboarding
> **NOT WIRED IN.** This directory is deliberately absent from `apps/kustomization.yaml`. > **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
> It lands **after** the storage workstream ships the **`longhorn-vm`** RWX StorageClass; the PM > (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
> then places `unifi` in the correct apps delivery *wave*. Nothing here should be reconciled yet. > lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
> or install cannot stall the app tier.
>
> **Status (verified 2026-08-26):**
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
## What this is ## What this is
@@ -18,12 +28,33 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
|---|---| |---|---|
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation | | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS | | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (60Gi). First real CDI http import on the cluster | | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, cloud-init that preps Podman + installs UOS | | `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress | | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record | | `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
| `90-httproute-optional.yaml` | **Optional, disabled** L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat) | | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
| `kustomization.yaml` | For `kubectl kustomize apps/unifi` validation only; excludes the optional HTTPRoute |
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
### Cloud-init delivery — two constraints learned the hard way
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
Secret deliberately.
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
### Node prerequisites this VM forced onto the cluster
- **`nonroot-devices: true`** in the k3s server config (`infra/ansible/roles/k3s_server`) — the
non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without
it: `blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
3-replica class made the import fail with insufficient storage.
## UOS install method (researched) ## UOS install method (researched)
@@ -72,38 +103,51 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy `https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`. (UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
## Open questions / risks (need a PM decision or a live spike) ## Open questions / risks
1. **Unattended install (HIGH).** The installer is a **signed, account-tied** URL that **rotates/ 1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
expires** (cannot be committed) and is documented as **interactive** (no official silent flag). account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
cloud-init preps the host fully and provides an `install-uos` helper + an optional baked-URL **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
auto-run that pipes `yes`. **Decision needed:** accept "operator runs `install-uos` at bringup" the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
vs. invest in a verified silent path. First bringup should be treated as a spike. **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
2. **CDI http import (MEDIUM).** First http/qcow2 CDI import on this cluster; no checksum on an http invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress 2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
CNP (included). import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
3. **Boot firmware (LOW).** Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot. Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
If not, add the `efi` firmware block (noted inline in `30-virtualmachine.yaml`). fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
4. **VIP `192.168.1.54` (LOW).** Confirm free, in-pool, and reserved in tofu (Gateway already holds 3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree. 4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
5. **Longhorn RWX prerequisite (BLOCKING).** `longhorn-vm` must exist with RWX/Block + migratable `192.168.1.53–.60` pool (`.53` is the shared Gateway).
before apply, else the DataVolume/VM stay Pending. 5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the 6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
installer and for optional remote access. installer and for optional remote access.
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
## Post-merge bring-up + verification ## Remaining bring-up
1. **Prereq:** `longhorn-vm` StorageClass live (RWX, Block, migratable). Confirm `192.168.1.54` free. Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
2. PM adds `apps/unifi` to the right wave in `apps/kustomization.yaml`; reconcile.
3. Watch the CDI import: `kubectl -n unifi get dv unifi-rootdisk -w` → `Succeeded`. 1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
4. VM starts: `kubectl -n unifi get vmi unifi` → `Running`. LB VIP: `kubectl -n unifi get svc` → 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
both show `192.168.1.54`. Kustomization (not an app wave).
5. **Install UOS** (if not auto-run): get the signed URL from ui.com, then in the guest 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
`sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos`. Access the guest via 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
`virtctl console unifi` / SSH once cloud-init adds a key. `192.168.1.54`.
6. Verify UI: browse `https://unifi.home.keithjackson.dev:11443`, complete first-run wizard. 5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54`. and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
complete the first-run wizard.
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN 8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
connects (green). Test guest portal / throughput if used. connects (green). Test guest portal / throughput if used.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS. 9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version. 10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
**Quick health check today:**
```bash
kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
```
ci/touched-requirements.sh
+42 -7
View File
@@ -16,9 +16,10 @@
# `conformance.sh --requirements ...` which runs exactly the tests carrying those tags. # `conformance.sh --requirements ...` which runs exactly the tests carrying those tags.
# #
# Detection = union of: # Detection = union of:
# 1. R-* tokens literally present in the diff of changed files — including the # 1. R-* tokens on the ADDED/REMOVED LINES of the diff, in non-Markdown files —
# authoritative `conformance.home.k3s/R-*` tags on changed tests, and the R-IDs # including the authoritative `conformance.home.k3s/R-*` tags on changed tests, and
# that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011). # the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
# Untouched lines and prose are NOT scanned; see changed_diff() for why.
# 2. A coarse path->requirement fallback for changes that don't self-declare IDs. # 2. A coarse path->requirement fallback for changes that don't self-declare IDs.
# The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for # The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
# untagged changes so nothing silently escapes the gate. # untagged changes so nothing silently escapes the gate.
@@ -45,6 +46,36 @@ changed_files() {
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
} }
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
# documentation.
#
# TWO scoping rules, both learned from the gate blocking merges it should not have:
#
# 1. Scan the DIFF, not whole file contents. Grepping whole files meant a change
# inherited every requirement those files had EVER mentioned, on lines the author
# never touched. Since some of those are legitimately uncovered, `--require-all`
# then failed and BLOCKED the merge over unrelated prose.
#
# 2. Skip `*.md`. A requirement ID in prose is a REFERENCE, not a declaration —
# explaining a requirement is not the same as changing behaviour against it.
# Without this, the decision log cannot record which requirements a change
# concerned without thereby claiming to have touched them, and the CI-fix commit
# that explains a mis-attribution re-triggers the very mis-attribution it fixes.
# Nothing escapes: authoritative declarations (`conformance.home.k3s/R-*` tags,
# Kyverno policies, ServiceDefinitions) all live in YAML, and the doc subtrees
# that matter (conformance/, schemas/, catalog/, clusters/, ci/) are covered by
# the path->requirement fallback regardless of file type.
#
# Removed lines count: deleting a test's requirement tag, or a policy's self-declared
# ID, touches that requirement just as much as adding one.
# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above). Three-dot to
# match changed_files(): changes on this branch since it diverged from base.
changed_diff() {
local base; base="$(resolve_base)"
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . ':(exclude)*.md' 2>/dev/null; then return; fi
git -C "$REPO_ROOT" diff -U0 "${base}" -- . ':(exclude)*.md' 2>/dev/null || true
}
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees # Coarse path -> requirement map. Deliberately conservative; extend as subtrees
# fill in. Left side is a path prefix (relative to repo root). # fill in. Left side is a path prefix (relative to repo root).
path_requirements() { path_requirements() {
@@ -74,12 +105,16 @@ main() {
while IFS= read -r f; do while IFS= read -r f; do
[ -n "$f" ] || continue [ -n "$f" ] || continue
reqs+=" $(path_requirements "$f")" reqs+=" $(path_requirements "$f")"
# Scan file contents for self-declared R-IDs (only if it still exists).
if [ -f "${REPO_ROOT}/${f}" ]; then
reqs+=" $(grep -hoE 'R-[A-Z]+-[0-9]+' "${REPO_ROOT}/${f}" 2>/dev/null | tr '\n' ' ' || true)"
fi
done <<< "$files" done <<< "$files"
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
reqs+=" $(changed_diff \
| grep -E '^[+-]' \
| grep -vE '^(\+\+\+|---)' \
| grep -hoE 'R-[A-Z]+-[0-9]+' \
| tr '\n' ' ' || true)"
# Normalize: split, unique, sort. Emit to stdout. # Normalize: split, unique, sort. Emit to stdout.
printf '%s\n' $reqs | sed '/^$/d' | sort -u printf '%s\n' $reqs | sed '/^$/d' | sort -u
} }
conformance/run.sh
+12 -3
View File
@@ -60,9 +60,18 @@ FORCE_NO_CLUSTER=0
# path-map keys to it, so editing any ci/ file (SPEC_PIN # path-map keys to it, so editing any ci/ file (SPEC_PIN
# adoption included) would false-block. # adoption included) would false-block.
# These are visible GAPs, not hidden — removing them from this list is the right move the # These are visible GAPs, not hidden — removing them from this list is the right move the
# moment WS-TEST adds their verifying tests. FOLLOW-UP (WS-TEST): the deeper fix is to make # moment WS-TEST adds their verifying tests.
# the path->requirement matcher stop keying on doc-header tokens so real requirements need #
# not be exempted; tracked as a harness debt item, not resolved here. # FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
# tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
# ci/touched-requirements.sh: it scans the added/removed lines of the diff, and skips *.md, so
# a requirement named in a header or in prose no longer counts as a declaration. Consequence
# to be aware of: a manifest whose HEADER declares the requirements it serves no longer
# contributes those IDs unless the change actually touches a line carrying them — the
# authoritative signal is the `conformance.home.k3s/R-*` label on a test, which still is.
# The PATH-MAP half remains: ci/touched-requirements.sh keys clusters/* and ci/* to
# R-ENV-02 + R-GITOPS-01 wholesale, which is why those two must stay exempted here. Tighten
# the path map (or scaffold the two requirements) and this list shrinks to R-STATE-02.
DOCUMENTED_GAPS="R-STATE-02 R-ENV-02 R-GITOPS-01" DOCUMENTED_GAPS="R-STATE-02 R-ENV-02 R-GITOPS-01"
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
infra/PHASE1-SCOPE.md
+15 -1
View File
@@ -1,8 +1,22 @@
# Phase 1 Scope — dev/test Cluster Bring-up (local KVM) # Phase 1 Scope — dev/test Cluster Bring-up (local KVM)
> ## ⚠️ HISTORICAL — DO NOT EXECUTE
>
> **Phase 1 is complete.** This document is the *original* scoping for the first dev cluster, which
> ran on **local libvirt/KVM VMs on the workstation** (`10.73.0.0/24`). That cluster was retired: it
> was constrained by the workstation's ~22 GiB RAM and 137 GiB disk, and the dev cluster was migrated
> to a **Proxmox VE** host (nodes `192.168.1.50/.51/.52`, 6 vCPU / 16 GiB each).
>
> Kept for the record — the exit criteria, sizing rationale, and risk analysis below are still the
> reasoning behind the current design. **Everything about libvirt, `virsh`, the `dmacvicar/libvirt`
> provider, `10.73.0.x`, and the "missing prereqs" table is superseded.**
>
> For the cluster as it exists today, read **[`PHASE1-STATUS.md`](PHASE1-STATUS.md)**.
**Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md` **Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md`
(Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux), (Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux),
0007 (SOPS). **Status:** scoped, awaiting operator go / prerequisite installs. 0007 (SOPS). **Status (as written, 2026-08-16):** scoped, awaiting operator go / prerequisite
installs. **Status now:** delivered, then superseded by the Proxmox migration.
## Objective & exit criteria (roadmap Phase 1) ## Objective & exit criteria (roadmap Phase 1)
A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC
infra/PHASE1-STATUS.md
+120 -48
View File
@@ -1,57 +1,129 @@
# Phase 1 — Execution Status & Resume Guide # Dev cluster — current state & reconnect guide
**Checkpoint: 2026-08-16.** The dev/test cluster is **built and running**. Phase 1 is paused before **Last verified: 2026-08-26.** Phase 1 is **complete and long since exited**; the cluster has been
the Flux/SOPS bootstrap (a deliberate decision — see "Remaining"). This doc is how a future session running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
picks up. reconnects to it and re-establishes the toolchain.
## DONE — a live 3-node HA k3s + Cilium cluster (T-1.1/1.2/1.3) > **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
- `tofu apply` created the `k3s-dev` NAT network (10.73.0.0/24) + 3 VMs (`k3s-dev-1/2/3`, > (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
6 GiB/4 vCPU, 20 G OS + 40 G data, host-passthrough) — all **running**. > now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
- cloud-init baselined the nodes; k3s HA (embedded etcd, 3 servers) + Cilium installed by Ansible. > or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and
- **Live health:** `kubectl get nodes` → **3/3 Ready**; `cilium status` → **OK**; CoreDNS up. > [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc).
### Fixes applied during execution (committed to main) ## The cluster
- `ansible.cfg`: `stdout_callback yaml` removed in ansible-core 2.20 → builtin default.
- `cilium-values`: `egressGateway` requires `bpf.masquerade: true` (+ `enableIPv4Masquerade`) or the
agent fatals — added.
## Toolchain reality (important for resume) | | |
The pinned **dev container** (`infra/devcontainer/`) holds flux/sops/age/kubectl/helm. BUT:
- **`tofu` runs on the HOST** — the `dmacvicar/libvirt` provider's read-write connect fails inside a
container (`Cannot find start time for pid`). The pinned binary was extracted to
**`~/.local/bin/tofu`** (OpenTofu 1.9.0). Run tofu from the host with `LIBVIRT_DEFAULT_URI=qemu:///system`.
- **`ansible` runs on the HOST** (pre-installed ansible-core 2.20; needs native network to the nodes).
## State locations (persisted on disk; gitignored)
| What | Path |
|---|---| |---|---|
| tofu state | `infra/tofu/terraform.tfstate` | | Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
| tofu vars (ssh pubkey) | `infra/tofu/terraform.tfvars` | | Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 16 GiB, Debian 12 (bookworm) |
| cluster kubeconfig | `infra/ansible/.cache/dev.kubeconfig` (API at 10.73.0.11:6443) | | k3s | **v1.31.5+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
| SSH key | `~/.ssh/home_network` (user `k3s`, NOPASSWD sudo) | | CNI | Cilium 1.16.5 (kube-proxy replacement, L2 announcements, egress gateway) |
| VMs / network | libvirt `qemu:///system` (`virsh list`, net `k3s-dev`) | | kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
| Storage | Longhorn 1.7.3 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |
Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium,
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`.
## Network path from the workstation (there is a NAT in the way)
The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two:
| Purpose | Workstation-side target | Forwards to |
|---|---|---|
| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` |
| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 |
| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host |
TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed.
## Access
**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates
against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the
kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`:
## Reconnect / manage
```bash
infra/cluster.sh status # VMs + kubectl get nodes
infra/cluster.sh start|stop # boot/graceful-shutdown the 3 VMs (state persists on disk)
eval "$(infra/cluster.sh kubeconfig)" # export KUBECONFIG
kubectl get nodes
ssh -i ~/.ssh/home_network [email protected]
``` ```
The VMs are **running** at checkpoint (~18 GiB RAM). `infra/cluster.sh stop` frees RAM; k3s/etcd/ kubectl config use-context k3s-oidc
Cilium recover on `start` (data on the disks). kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
```
## REMAINING Phase 1 `oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so
- **T-1.4 Flux + SOPS bootstrap — Option A (decided):** start the host Gitea (`bootstrap/gitea/`), this path needs an interactive desktop.
push this repo, `flux install`, apply the `flux-system` GitRepository + age Secret — but **keep the
`platform` and `apps` Kustomizations SUSPENDED**. This proves Flux reconciles from the host Gitea
(Phase-1 exit) WITHOUT unleashing the incomplete Phase-0 platform stubs on the working cluster.
- **T-1.5** destroy/rebuild proof (`tofu destroy` a node → `apply` → re-join), then flip
`CLUSTER_AVAILABLE=true` so conformance L2/L3 runs.
## Phase 2 prerequisite (before un-suspending `platform/`) **kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from
The Phase-0 `platform/` HelmReleases are stubs. **Do NOT let Flux reconcile them onto this cluster `/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and
until:** `platform/cilium` `k8sServiceHost` is set to **10.73.0.11** (parity with the Ansible install, add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the
docs/02) and the REPLACE-ME values / secrets (Cloudflare token T-2.4p, etc.) are filled — otherwise gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need
Flux would `helm upgrade` Cilium with a broken value and could break the running CNI. this, since they cannot complete a browser login.
**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 [email protected]` (user `k3s`, NOPASSWD sudo).
Ports 2251/2252 for nodes 2/3.
## Toolchain — use the dev container, not the host
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate
toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
```bash
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
infra/devcontainer/dev.sh # interactive shell at /work
```
Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket,
which is what broke the old `dmacvicar/libvirt` provider in a container.
`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring —
**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase
file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
check runs instead of silently skipping.
> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`,
> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl`
> therefore needs an explicit `DEV_KUBECONFIG=<path to a break-glass cert kubeconfig>` — the OIDC
> context cannot work there anyway (no browser). Not yet fixed.
## Off-repo state you must not lose
| What | Where |
|---|---|
| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret |
| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` |
| Node SSH key | `~/.ssh/home_network` |
| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state |
| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes |
| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` |
## Health check
```bash
kubectl get nodes # 3/3 Ready
flux get kustomizations # flux-system, platform, platform-config,
# apps-wave-0/1/2, apps-vms — all Ready
kubectl get helmreleases -A # all Ready
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
kubectl get vmi -A # unifi Running
```
## Flux's git source is inside the cluster — mind the loop
`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**:
the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes
through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea,
the Gateway, CoreDNS and the wildcard cert all being up.
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap
Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that
is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept.
**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy.
There is no separate "apply" step and no GitHub remote.
infra/README.md
+28 -3
View File
@@ -1,4 +1,29 @@
# infra/ — Phase 1 node provisioning (local KVM) # infra/ — node provisioning and the toolchain
OpenTofu (libvirt) + cloud-init + Ansible bring up the 3-node k3s/Cilium cluster. OpenTofu (**Proxmox VE**) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
See PHASE1-SCOPE.md. Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md. Spec: `../home-network-k3s/docs/02-environments-and-provisioning.md`.
| Path | What |
|---|---|
| [`tofu/`](tofu/) | The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (`bpg/proxmox` provider) |
| [`cloud-init/`](cloud-init/) | First-boot baseline shipped as a Proxmox user-data snippet |
| [`ansible/`](ansible/) | The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
| [`devcontainer/`](devcontainer/) | The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm **and the whole CI gate toolchain**) |
| [`cluster.sh`](cluster.sh) | **RETIRED** — drove the old libvirt VMs; does not work against Proxmox |
| [`PHASE1-STATUS.md`](PHASE1-STATUS.md) | **Current** state of the cluster and how to reconnect to it |
| [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) | Historical Phase-1 scoping doc (libvirt era — superseded) |
## Where the cluster actually lives
The dev cluster is **not** on this workstation. It runs on a Proxmox VE host; the nodes sit on the
Proxmox LAN and are reached across a pfSense NAT:
| | |
|---|---|
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52` — 6 vCPU, 16 GiB, Debian 12, k3s v1.31.5 |
| kube-API | kube-vip VIP `192.168.1.49:6443`, reached from the workstation via NAT `192.168.2.50:6443` |
| Node SSH | pfSense NAT `192.168.2.102:2250/2251/2252` → node `:22` (Ansible only) |
| Ingress VIP | `192.168.1.53` (Cilium L2, shared Gateway); `.53–.60` reserved for LB VIPs |
The earlier libvirt/KVM cluster on `10.73.0.0/24` is **retired**. Any doc or command referencing
`virsh`, `qemu:///system`, or `10.73.0.x` for the cluster is historical.
infra/cluster.sh
+13 -2
View File
@@ -1,6 +1,17 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# infra/cluster.sh — manage the local dev k3s cluster (VMs + kubeconfig). Convenience for # infra/cluster.sh — RETIRED (libvirt era). DO NOT USE.
# resuming Phase 1 development. Uses host virsh + kubectl (see infra/PHASE1-STATUS.md). #
# This managed the ORIGINAL dev cluster: 3 libvirt/KVM VMs on the workstation, on the
# 10.73.0.0/24 NAT network. That cluster no longer exists — the dev cluster was migrated to
# Proxmox VE (nodes 192.168.1.50/.51/.52) and there are no local `k3s-dev-*` domains for
# virsh to start or stop. It is kept only as a record of the retired lifecycle.
#
# What to use instead (see infra/PHASE1-STATUS.md):
# VM power — the Proxmox UI / API on the Proxmox host, or `tofu` in infra/tofu
# kubeconfig — ~/.kube/config context `k3s-oidc` (Keycloak SSO), or the break-glass
# cert kubeconfig from the primary node
# node access — ssh -p 225{0,1,2} [email protected] (pfSense NAT)
# cluster health — kubectl get nodes && flux get kustomizations
set -euo pipefail set -euo pipefail
export LIBVIRT_DEFAULT_URI=qemu:///system export LIBVIRT_DEFAULT_URI=qemu:///system
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
infra/devcontainer/README.md
+68 -36
View File
@@ -1,51 +1,83 @@
# Dev toolbox (containerized baseline) # Dev toolbox (containerized baseline)
The Phase 1 prerequisites are **not all Python** (OpenTofu, `age`, `sops`, `flux` are Go binaries; The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`,
only Ansible is Python). Per the operator's rule, the toolchain is therefore a **pinned dev `kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
container** baked with everything needed to drive the local KVM cluster — no host pollution, one rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible
reproducible baseline. baseline, and the *same* binaries CI uses.
**This is the supported way to run anything in this repo.** Don't install the toolchain on the host.
## What's inside (pinned in `versions.env`) ## What's inside (pinned in `versions.env`)
OpenTofu, `age` + `age-keygen`, `sops`, `flux`, `kubectl`, `helm`, `ansible-core`, `libvirt-client`
(so the tofu `dmacvicar/libvirt` provider can talk to the host libvirt), `qemu-img`, cloud-init seed | Group | Tools |
tooling (`cloud-utils`/`genisoimage`), `openssh-clients`, and common utilities. |---|---|
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
| GitOps / cluster | `flux`, `kubectl`, `helm` |
| Secrets | `sops`, `age` + `age-keygen` |
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
The conformance-tool versions are kept in lockstep with `ci/versions.env` so
`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records
the resolved set at `/etc/dev-toolchain-versions`.
## Use ## Use
```bash ```bash
cd infra/devcontainer cd infra/devcontainer
./dev.sh build # build the image (needs network: base image + pinned binaries) ./dev.sh build # build the image (needs network)
./dev.sh # build if needed + open a shell in /work (the repo) ./dev.sh # build if needed + shell in /work (the repo)
./dev.sh run tofu -version ./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
./dev.sh run sops -d path/to.sops.yaml
./dev.sh run tofu -chdir=/work/infra/tofu plan
``` ```
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman. `ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
## How it reaches the host KVM ## What `dev.sh` mounts, and why each one matters
`dev.sh` runs the container with:
- `--network host` — so Ansible can SSH to the cluster VMs on the `k3s-dev` NAT subnet
(10.73.0.0/24) and `kubectl`/`flux` can reach the API server.
- `-v /run/libvirt:/run/libvirt` + `LIBVIRT_DEFAULT_URI=qemu:///system` — so OpenTofu/virsh drive
the host libvirt daemon (the socket is world-writable; no group change needed).
- `--security-opt label=disable` — lets the container use the host libvirt socket without
relabeling it (SELinux hosts).
- `-v $REPO:/work` and `-v ~/.ssh:/root/.ssh:ro` — the repo and your SSH keys.
## Known limitation — run `tofu` and `ansible` on the HOST, not in the container - `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
Discovered during Phase 1 bring-up (see `../PHASE1-STATUS.md`): `kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API
- **OpenTofu + `dmacvicar/libvirt`** fails *inside* the container — its read-write `qemu:///system` (`192.168.2.102:8006`) on the host's network.
connect hits a polkit/PID-identity path (`Cannot find start time for pid`) that a mounted socket - `-v $REPO:/work` (workdir) — the repo.
can't satisfy across the PID namespace (`--pid=host` didn't help). **Run the pinned tofu on the - `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the
host** (extract once: `podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/`; private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with
then `LIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...`). the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable.
- **Ansible** talks to the cluster VMs over SSH; simplest to run the pre-installed host `ansible-core` - `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`).
with native networking to the `k3s-dev` NAT subnet. - `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted
- The container remains the baseline for **flux/sops/age/kubectl/helm** (used from inside with `group_vars` decrypt unattended.
`--network host` once the cluster exists). - `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when
it exists beside this one. Without it `ci/spec-immutability.sh` honestly *skips*
rather than passing — mount it, or that gate silently verifies nothing.
- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling.
## tofu and ansible now run **inside** the container
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a
container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and
run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which
talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs
SSH, which `--network host` provides.
So: run everything in here. The host needs only podman (or docker).
## kubectl inside the container — known gap
`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path
that no longer exists**, and does not mount `~/.kube`. Override it explicitly:
```bash
DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
```
Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser,
which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both
access paths. (Not yet fixed in `dev.sh`.)
## Notes ## Notes
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env` - Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
(one line) to the current stable and rebuild. The image records the resolved set at and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`.
`/etc/dev-toolchain-versions`. - **Secrets never enter the image** — they arrive only as the bind mounts listed above.
- Secrets never enter the image. The SOPS **age private key** is generated at bootstrap and kept - This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
out of git (`.gitignore` guards `age.key`). [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md).
- This container is the toolchain, not the cluster. It provisions the 3 KVM nodes (T-1.1–T-1.4);
the cluster itself runs on libvirt on the host.
infra/tofu/README.md
+23 -10
View File
@@ -2,12 +2,15 @@
Provisions the "Create" lifecycle stage (docs/02): 3 cloud-init'd VMs on the Provisions the "Create" lifecycle stage (docs/02): 3 cloud-init'd VMs on the
**Proxmox VE** box, each with an imported OS disk and a blank Longhorn data disk. **Proxmox VE** box, each with an imported OS disk and a blank Longhorn data disk.
**Authoring only — nothing here has been applied.** (Migrated from the old (Migrated from the old memory-constrained libvirt host; provider is now `bpg/proxmox`.)
memory-constrained libvirt host; provider is now `bpg/proxmox`.)
**APPLIED AND LIVE.** These three VMs are the running dev cluster — `k3s-dev-1/2/3`, k3s
v1.31.5, carrying the whole platform. Treat any `tofu apply/destroy` here as an operation on
production-for-dev, and note the state file caveat below.
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate). Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate).
## What it builds (defaults = VERIFIED Proxmox targets) ## What it builds (defaults = the LIVE cluster, verified 2026-08-26)
| Resource | Value | | Resource | Value |
|---|---| |---|---|
@@ -16,11 +19,19 @@ Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate).
| Image/snippet datastore | `local` (dir; content = iso + **snippets**) | | Image/snippet datastore | `local` (dir; content = iso + **snippets**) |
| Bridge | `vmbr0` (no VLAN) | | Bridge | `vmbr0` (no VLAN) |
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52`, gw `192.168.1.1`, DNS `192.168.1.18` | | Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52`, gw `192.168.1.1`, DNS `192.168.1.18` |
| Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt) | | Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt — real KVM confirmed in-guest) |
| OS disk | 40 GiB, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image | | OS disk | **64 GiB**, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image |
| Data disk | 100 GiB, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init) | | Data disk | **128 GiB**, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init; Longhorn reports ~125 GiB usable per node) |
| Cloud-init | native `initialization` (static IP + DNS) + user-data **snippet** for the baseline | | Cloud-init | native `initialization` (static IP + DNS) + user-data **snippet** for the baseline |
| Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs) | | Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs). In use: `.53` shared Gateway, `.54` UniFi |
## ⚠️ State file
`terraform.tfstate` is gitignored and lives only on the workstation that ran the apply. On a
**fresh workstation it is absent** — and a `tofu apply` with empty state will try to *create* three
VMs that already exist. Recover the state file (or `tofu import` the existing VMs) before running
anything that plans changes. `tofu plan` against empty state is itself harmless, but read the plan:
"3 to add" means you are missing state, not that the cluster is gone.
All sizing/IPs/counts/placement are variables — see `variables.tf`. All sizing/IPs/counts/placement are variables — see `variables.tf`.
@@ -45,9 +56,11 @@ All sizing/IPs/counts/placement are variables — see `variables.tf`.
without `https://`, the provider talks HTTP and eats a 301. Either fix the env without `https://`, the provider talks HTTP and eats a 301. Either fix the env
file to include the scheme, or export `TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT"` file to include the scheme, or export `TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT"`
so tofu normalizes it (prepends `https://`). so tofu normalizes it (prepends `https://`).
3. **pfSense SSH NAT** (network-dependent): the PM reaches these nodes and the k3s 3. **pfSense NAT** — **DONE, live.** The workstation reaches the nodes and the kube-API
API over a pfSense NAT that is not yet created; the Ansible inventory carries a across it: SSH `192.168.2.102:2250/2251/2252` → `192.168.1.50/.51/.52:22`, and kube-API
note to finalize `ansible_host` once the NAT rule exists. `192.168.2.50:6443` → kube-vip VIP `192.168.1.49:6443`. `infra/ansible/inventory/hosts.ini`
is **hand-maintained** for this mapping — do **not** regenerate it from
`tofu output -raw ansible_inventory`, which knows nothing about the NAT and would overwrite it.
## Auth (env only — never in tofu vars/state) ## Auth (env only — never in tofu vars/state)
The provider reads these from the environment; the **API token is never a tofu The provider reads these from the environment; the **API token is never a tofu
infra/tofu/variables.tf
+1 -1
View File
@@ -74,7 +74,7 @@ variable "node_count" {
} }
} }
# --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 40 GiB OS + 100 GiB data) -- # --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 64 GiB OS + 128 GiB data) --
variable "vcpu" { variable "vcpu" {
description = "vCPUs (cores) per node. Generous — the box has plenty." description = "vCPUs (cores) per node. Generous — the box has plenty."