diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 76bbc7e..2bd9f67 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -922,3 +922,184 @@ R-TLS-01/03 GREEN** (rewrote the probe for LE-prod system-trust + split-horizon LESSON: heavy sustained node load (TCG-emulated VMs) can degrade Cilium datapath -> intermittent LB-VIP reachability; a restart clears it. NOTE: after a host reboot, restart the bootstrap Gitea container (`bash bootstrap/gitea/bootstrap-gitea.sh`) - it is Flux's source and does not auto-start; the k3s VMs auto-recover. + +### BACKFILL — Proxmox migration, app tier, wave split, UniFi VM, kubectl SSO (2026-08-25 → 08-26) +**Process note first:** this log had gone stale at `1eb2cb7` (2026-08-25 AM) while `main` advanced +**~85 commits** to `3472f1d`. Everything below is reconstructed from those commits and **verified +against the live cluster on 2026-08-26**. The lesson is the obvious one: the log is the PM's +authoritative record and a two-day gap in it cost a session's worth of re-derivation. Append per +work item, not per session. + +**1. INFRA — the dev cluster MIGRATED off libvirt onto Proxmox VE.** The workstation-hosted libvirt +cluster (3 VMs on `10.73.0.0/24`, 6 GiB each) was RAM/disk-starved — the KubeVirt work was OOMing +nodes (`971c730` had to temp-disable KubeVirt just to relieve pressure). Provisioning was ported +`dmacvicar/libvirt` → **`bpg/proxmox`** (`53ca698`), cloud-init moved from a netplan/meta-data seed +ISO to a Proxmox **user-data snippet** and then to API-native cloud-init with the rich baseline +pushed into the Ansible `node_baseline` role (`39123d7`, `a2392a4`). Nodes are now +**`192.168.1.50/.51/.52`, 6 vCPU / 16 GiB / 64 GiB OS + 128 GiB Longhorn data** (`8a8f28b`), Debian +12, `cpu_type=host` for nested KVM. Gotchas paid for along the way: the base image must be +downloaded as **import** content-type for `import_from` (`fca8d8f`); the Debian *genericcloud* +image kernel-panics on first boot under bpg — use the **generic** image + serial console +(`28d200c`, upstream bpg#1639); `qemu-guest-agent` must be enable-only when tofu leaves the virtio +channel off, or the baseline play fails (`1004c8e`). + +**2. NETWORK — pfSense NAT, then kube-vip, then the tunnel died.** The workstation is not on the +cluster LAN. First cut routed Ansible over a pfSense SSH NAT (`192.168.2.102:225x` → node `:22`) +and decoupled `k3s_node_ip` from `ansible_host` (`1ff2531`) — `inventory/hosts.ini` is therefore +**hand-maintained and must never be regenerated** from `tofu output ansible_inventory`. The +kube-API initially needed an SSH tunnel (`0037fb4`). That was retired by **kube-vip**: a +control-plane HA VIP at **`192.168.1.49`**, added to the k3s `tls-san` list so TLS validates +through the NAT via `tls-server-name` (`60a58f0`, `81ef782`). Controller access is now direct: +`https://192.168.2.50:6443` → VIP. Gateway/ingress VIP moved to **`192.168.1.53`** on the Proxmox +LAN, and `GATEWAY_VIP` + the split-horizon CoreDNS answer + the DNS conformance tests were all +repointed (`604552e`, `63b87ff`). + +**3. COLD-START DEADLOCK, again — and the durable lesson.** Bringing the platform up on the new +cluster re-hit the Phase-6 shape: `oauth2-proxy` sits in the `platform` `wait:true` tier but needs +Keycloak + trusted TLS + split-horizon DNS, so while it Failed the whole tier stayed not-Ready and +`platform-config` (which applies the fix) could never reconcile. Broken the same way as before — +temp-disable `oauth2-proxy` and its orphaned `platform-config` refs (`2fea1e2`, `efd6dbe`), let +platform go Ready, then re-enable (`e6b3a03`). KubeVirt/CDI re-enabled once the 16 GiB nodes +existed (`21669da`). **The FOLLOW-UP from Session 7 is still open and is now twice-proven: move +`oauth2-proxy` to a post-platform tier (`dependsOn: platform-config`) so a leaf forward-auth +service cannot gate its own dependencies.** + +**4. FLUX SOURCE — now the in-cluster Gitea.** Bootstrap source repointed (`fff4499`), Gitea fixed +to mount its PVC at `/data` with a headless-install bootstrap (`73c4475`), then +`GitRepository/flux-system` cut over to **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`** +(`d99d171`). Note the loop this creates: Flux's source is an app Flux deploys, reached via +split-horizon DNS → Gateway VIP. Stable in practice (Gitea's Deployment/PVC survive reboots +without Flux), but if Gitea is ever *lost* rather than restarted, recovery is the host bootstrap +Gitea in `bootstrap/`. `git push origin main` **is** the deploy — there is no other apply step. + +**5. CI — the merge gate is now REAL and in-cluster.** Gitea Actions enabled with an in-cluster +`act_runner` + DinD, running the static gate as the single required check on `main` (`bc4519b`). +Fixes: kyverno CLI arch is `x86_64` not `amd64`, and the job must wait for DinD certs (`79e7f92`); +`make` masked spec-immutability's exit 3 as a 2, so `gate.sh` calls the script directly +(`0f75fa2`); `upload-artifact@v4` is unsupported by Gitea Actions (`67bc317`). The **whole test +toolchain is now baked into the devcontainer** (`bb48f67`) so `make -C ci gate` on a laptop is +byte-identical to CI (R-TEST-02). Kyverno itself needed two real fixes: the `default-deny-shape` +policy's scalar-array pattern was **inverted** and silently accepted Ingress-only policies — now +expressed via `deny.conditions` requiring both (`1b0848c`); and the fail-closed admission +controller was crashlooping into 502s until its resources were raised and dev replicas set to 2 +(`28a34c4`). Chainsaw parallelism dropped 4→2, then heavy L2/L3 tests serialized, to stop dev etcd +overload cascades (`9f52f19`, `948c390`). + +**6. APP TIER — 10 services onboarded through the renderer.** gitea, plex, sonarr, radarr, +qbittorrent, jackett (`b76fb7b`…`8392b52`), then seerr (`b6f72ea`), nextcloud (`4be6752`) and +semaphore (`77496f9`) — the last two being the **first CNPG-DB + OIDC-target patterns**. Renderer +bugs found by real onboarding, each fixed in the renderer rather than by hand-patching output: +world/internet egress allow emitted per egress-class (`7e8ae47`); **reciprocal ingress-allow** for +declared dependencies, without which a declared dep silently did not connect under default-deny +(`14beb85`, R-NET-02); east-west selectors **namespace-pinned** so they match the right pods +(`b871877`); `Recreate` strategy for Deployments mounting RWO PVCs, else rollouts deadlock +(`d32b5c4`). ServiceDefinition schema gained `env`/`envFrom`/`lifecycle`, retiring the bespoke +gitea bootstrap patch (`c8dd4f3`). Semaphore cost two root-causes: `fsGroup` via schema-1.3.0 +`podSecurityContext` for non-root PVC write (`28498f1`), and **`SEMAPHORE_PORT` being shadowed by +kubelet service-link env injection** (`41d696e`) — a trap for any app whose config env name +collides with a Service name. + +**7. APP TIER TOPOLOGY — monolithic `apps` split into WAVES.** One `wait:true` Kustomization over +all ~130 app resources meant one crashloop blocked every app and every new revision (it bit twice). +Split into `apps-wave-0/1/2`, each `dependsOn` the previous, wave assignment computed by +`catalog/render.py::compute_waves()` from the declared dependency graph. Design + the data-loss +cutover analysis are in **`DESIGN-waves.md`**. The cutover was executed as designed: `prune:false` +on the old `apps` first (`1bfb0db`), then the split (`36a1a4e`) so the waves **adopt** by identity +instead of delete/recreate. **VERIFIED 2026-08-26:** `nextcloud-db` carries +`kustomize.toolkit.fluxcd.io/name=apps-wave-0`, the CNPG Clusters retain their ORIGINAL ages +(22h/21h — not recreated), and `prune:true` is restored on the waves. No data was lost. + +**8. PHASE 5 — storage for real VMs, and the UniFi VM.** CSI snapshots + smart-clone + a migratable +RWX `longhorn-vm` StorageClass landed to make KubeVirt **live migration** possible (`3586769`); +two KubeVirt alerts were plain wrong and firing falsely — a broken http CDI ServiceMonitor causing +`CDIOperatorDown` (`6a7e9d1`) and `KubeVirtControlPlaneDown` using a non-existent metric with the +wrong job label (`b57b3e9`). **UniFi OS Server** was then onboarded as a VM on a deliberately +**non-gating** track (`317b3f0`): `apps/vms/` reconciled by `apps-vms` (`wait:false`, +`dependsOn: platform-config`), because a VM's image import + in-guest install is legitimately +long-not-Ready and must never gate the app tier. Four cluster-level truths came out of it: +- **`nonroot-devices: true`** on the k3s servers — the non-root CDI importer must open a raw Block + device k8s never chowns for it (`df5979f`). +- **`longhorn-scratch`**, a single-replica class for CDI scratch; the 3-replica default failed the + import with insufficient storage (`7a3da96`). +- **KubeVirt caps inline `userData` at 2048 bytes.** The rich cloud-init exceeded it; a + `userDataSecretRef` was tried and reverted; shipped state is a **minimal ~1.1 KB inline** + payload (`e99e3d1`, `760084e`, `ab59202`). The `install-uos` helper did not survive the cut. +- **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux envsubst rewrote `${...}` inside + the cloud-init and corrupted it (`da1f833`). Do not restore it for consistency with the waves. + +**9. PHASE 6 — kubectl login via Keycloak SSO (`3472f1d`).** A **public** `kubernetes` client with +PKCE (no secret in any kubeconfig), a groups mapper, kube-apiserver OIDC flags via the k3s_server +role, and `k8s-oidc-rbac.yaml` binding `oidc:platform-admins` → `cluster-admin`, +`oidc:viewers` → `view`. The cert kubeconfig is untouched and remains a fully independent +**break-glass** path. **VERIFIED LIVE:** `kubectl auth whoami` returns +`oidc:` with `[oidc:platform-admins]` — so the API-server flags are active; the rolling +restart the commit message deferred has since happened. + +**LIVE STATE VERIFIED 2026-08-26 (fully green):** 3/3 nodes Ready (k3s v1.31.5); **7/7 Flux +Kustomizations Ready** at `3472f1d` (flux-system, platform, platform-config, apps-wave-0/1/2, +apps-vms); **10/10 HelmReleases Ready**; zero failing pods; 3 CNPG Clusters healthy (keycloak, +nextcloud, semaphore); UniFi VMI Running with guest agent connected (Ubuntu 24.04.4, +LiveMigratable). Conformance scaffold coverage **36/51 (70.6%)**. + +**GATE WAS RED — two independent defects found and FIXED (branch `docs/reconcile-with-reality`).** +`main` had been failing the merge gate since the UniFi commits landed. Two separate causes: + +1. **yamllint `commas` errors** in the hand-authored `apps/vms/unifi/` manifests + (`30-virtualmachine.yaml`, `40-networkpolicy.yaml`, `50-service-l4.yaml` — column-aligned inline + flow mappings). Cosmetic, but `lint` blocks the merge, so the VM work went in without a passing + gate. Fixed by normalizing the spacing; `kubectl kustomize apps/vms` renders **byte-identical** + before and after. The lint config was deliberately NOT relaxed — loosening a rule to make failing + files pass is moving the goalposts. + +2. **`ci/touched-requirements.sh` mis-attributed requirements** — the real defect, and the one + previously logged as "FOLLOW-UP (WS-TEST): fix the matcher so real requirements need not be + exempted". The script's own header says it scans "R-* tokens literally present in the **diff**", + but it actually grepped **whole file contents**. So any edit to a long-lived file pulled in every + requirement that file had ever mentioned in prose: appending one entry to **this log** dragged in + `R-SEC-01`/`R-SEC-03`/`R-TEST-06`, and touching `infra/tofu/` dragged in `R-ENV-01`/`R-PLAT-01`. + Several are legitimately uncovered, so `--require-all` then **BLOCKED the merge on lines the + author never touched** — i.e. the PM could not append to the decision log without failing CI. + Fixed with two scoping rules: scan the added/removed lines of the diff (as the header always + claimed), and **skip `*.md`** — an ID in prose is a *reference*, not a declaration. Without the + second rule the log cannot record which requirements a change concerned without thereby claiming + to have touched them, and the very commit that explains a mis-attribution re-triggers it. + Removed lines still count (deleting a `conformance.home.k3s/R-*` tag touches that requirement too). + **Known consequence, accepted:** a manifest whose *header* declares the requirements it serves + (e.g. `platform/keycloak/keycloak-realm-import.yaml`) no longer contributes those IDs unless the + change touches a line carrying them. That is the intended semantics — the harness debt was + written as "stop keying on **doc-header tokens**" — and the authoritative signal, a test's + `conformance.home.k3s/R-*` label, is still detected. Verified: a range spanning conformance and + platform YAML changes still resolves R-NET-01, R-OBS-02/04, R-STATE-01/02 and R-TEST-01..04. + **This is a gate-semantics change — flagged for review.** It is a tightening of attribution, not + a weakening: no requirement is exempted, `DOCUMENTED_GAPS` is untouched, authoritative + declarations (test tags, Kyverno policies, ServiceDefinitions) all live in YAML on lines a real + change necessarily touches, and the doc subtrees that matter (`conformance/`, `schemas/`, + `catalog/`, `clusters/`, `ci/`) stay covered by the path→requirement fallback regardless. The alternative fixes were worse: + adding five more `DOCUMENTED_GAPS` genuinely weakens the gate, and scrubbing requirement IDs out + of prose (precedent `f9e9640`) would mean vandalizing this log's own history to appease a matcher. + +Gate after both fixes, run in the devcontainer: spec-immutability ✅ (spec pinned at `b633b7f`, +matching the sibling repo), lint ✅, Kyverno L0 ✅, L1 render 3/3 ✅ — **MERGE GATE: PASS**. + +**OPEN / NEXT (carried forward):** +1. **Install UniFi OS Server in the guest** — the VM runs bare Ubuntu; `:11443` answers nothing + (CNP permits `cluster`→`:11443`, so it is genuinely not listening). Install is interactive with + a rotating signed URL — needs an operator decision (hand-install vs. a verified silent path). +2. **`R-STATE-02`** — the last safety-critical requirement with no L3 test (Longhorn node-loss). + Currently exempted via `DOCUMENTED_GAPS`. +3. **oauth2-proxy → post-platform tier** (twice-proven cold-start deadlock). +4. **OQ-BKP** — off-cluster backup target; gates Velero, CNPG DR depth, and `R-STATE-03`. +5. **OQ-2** — the `bulk` StorageClass exists but has `server/share: REPLACE-ME` and provisions + nothing; plex/arr/qbittorrent ship config-only until it is resolved. +6. **image-automation** still authored-but-disabled (needs a Gitea write-cred). +7. **Coverage matcher, remaining half.** The content-scan half is fixed (see above). The coarse + `path_requirements` half still over-matches: any `clusters/*` edit claims `R-ENV-02` + + `R-GITOPS-01`, which is why both remain in `DOCUMENTED_GAPS`. Tighten the path map, or scaffold + those two, and drop the exemptions (WS-TEST). +8. **`dev.sh` KUBECONFIG** defaults to the dead libvirt-era path + `infra/ansible/.cache/dev.kubeconfig` and `~/.kube` is not mounted — in-container kubectl needs + an explicit `DEV_KUBECONFIG` pointing at a break-glass cert kubeconfig (OIDC needs a browser). +9. **tofu state** is absent on a fresh workstation; a `tofu apply` with empty state would try to + recreate the three live nodes. Recover the state file or `tofu import` before planning changes. +10. **Docs hygiene.** This log going two days stale is what made the whole reconciliation necessary. + Append per work item, not per session. diff --git a/README.md b/README.md index fbb5ff1..7fd74c1 100644 --- a/README.md +++ b/README.md @@ -16,16 +16,62 @@ specification** at [`../home-network-k3s`](../home-network-k3s). ## Layout ``` -platform/ platform services (Flux HelmReleases/Kustomizations): cilium, longhorn, - cert-manager, externaldns, gateway, monitoring, velero, kubevirt, - image-automation, keycloak, oauth2-proxy, kyverno/ -apps/ application workloads, one dir per catalog service -catalog/ ServiceDefinition entries (the onboarding surface) -schemas/ ServiceDefinition JSON Schema + renderer contract -clusters/ Flux entrypoints + dev/ and prod/ overlays -conformance/ Chainsaw behavior/conformance suite + harness + coverage report -.gitea/ Gitea Actions CI (lint · schema · Kyverno L0 · Chainsaw trigger · merge gate) +infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the + pinned dev container that carries the whole toolchain +bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea +platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium, + longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns, + vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt +platform-config/ platform CONFIGS — the custom resources of the CRDs installed above + (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs). + Split out because CRs and the CRDs they use cannot apply in one pass. +apps/ application tier, reconciled as dependency-ordered WAVES: + wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md) + vms/ KubeVirt VirtualMachines (non-gating tier) +catalog/ ServiceDefinition entries (the onboarding surface) + render.py +schemas/ ServiceDefinition JSON Schema + renderer contract +clusters/ Flux entrypoints + dev/ and prod/ overlays +conformance/ Chainsaw behavior/conformance suite + harness + coverage report +ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision) +.gitea/ Gitea Actions CI running that gate on an in-cluster runner ``` -Current phase: **Phase 0 — Foundations** (repo, CI, ServiceDefinition schema, conformance harness, -Kyverno L0 gate). No cluster exists yet; Phase 1 (cluster bring-up) is not authorized. +## Status + +Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not +open** — `clusters/prod` is a design-only overlay. + +| Phase | Delivered | +|---|---| +| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI | +| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS | +| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS | +| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch | +| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled | +| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** | +| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate | + +**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap +(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`; +the full report is `conformance/_artifacts/coverage/coverage.md`. + +The running narrative — decisions, root causes, incidents, follow-ups — is +[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record. + +## Working on this repo + +Everything the gate needs is baked into the pinned dev container — **do not install the +toolchain on the host**: + +```bash +infra/devcontainer/dev.sh build # once +infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs +infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible) +``` + +The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only +so the spec-immutability check can actually run. See +[`infra/devcontainer/README.md`](infra/devcontainer/README.md). + +A green run prints `MERGE GATE: PASS`. The L2/L3 conformance layers stay `SKIPPED` unless +`CLUSTER_AVAILABLE=true` — the static gate never pretends to have run them. diff --git a/apps/vms/unifi/10-default-deny.yaml b/apps/vms/unifi/10-default-deny.yaml index 603a53c..31d28cb 100644 --- a/apps/vms/unifi/10-default-deny.yaml +++ b/apps/vms/unifi/10-default-deny.yaml @@ -1,5 +1,5 @@ --- -# STAGING / DRAFT — see namespace.yaml header. +# LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header). # # Default-deny baseline (R-NET-01): deny all ingress + egress, allow ONLY DNS to the # in-cluster resolver. Identical shape to the rendered pihole/app baseline. The VM's diff --git a/apps/vms/unifi/20-datavolume.yaml b/apps/vms/unifi/20-datavolume.yaml index 71d15e5..c89ef07 100644 --- a/apps/vms/unifi/20-datavolume.yaml +++ b/apps/vms/unifi/20-datavolume.yaml @@ -1,5 +1,5 @@ --- -# STAGING / DRAFT — see namespace.yaml header. +# LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header). # # CDI DataVolume — the UniFi VM boot disk. This is the FIRST real CDI *http* import on # this cluster (the vm-demo/conformance path imports a containerdisk from a *registry*; @@ -9,7 +9,7 @@ # STANDALONE DataVolume (NOT a dataVolumeTemplate) so the disk OUTLIVES `kubectl delete vm` # and survives destroy/rebuild (R-STATE-01) — same durability contract as vm-demo. # -# STORAGE = longhorn-vm (the RWX/migratable StorageClass the storage workstream is adding): +# STORAGE = longhorn-vm (the RWX/Block/migratable StorageClass; live in platform/longhorn): # - accessModes: ReadWriteMany + volumeMode: Block => REQUIRED for KubeVirt LIVE MIGRATION # (evictionStrategy: LiveMigrate in the VM). A ReadWriteOnce disk cannot live-migrate. # - 40Gi: the Ubuntu cloud image virtual size is ~3.5Gi; cloud-init growpart/growroot diff --git a/apps/vms/unifi/30-virtualmachine.yaml b/apps/vms/unifi/30-virtualmachine.yaml index 75c2dd4..988aa7a 100644 --- a/apps/vms/unifi/30-virtualmachine.yaml +++ b/apps/vms/unifi/30-virtualmachine.yaml @@ -65,14 +65,14 @@ spec: # these and (b) surfaces them as containerPorts on the virt-launcher pod so the # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them. ports: - - { name: inform, port: 8080, protocol: TCP } # device inform / adoption - - { name: ui-legacy, port: 8443, protocol: TCP } # legacy Network UI (still served) - - { name: ui, port: 11443, protocol: TCP } # UOS web UI (modern default) - - { name: portal-http, port: 8880, protocol: TCP } # guest portal HTTP - - { name: portal-https, port: 8843, protocol: TCP } # guest portal HTTPS - - { name: throughput, port: 6789, protocol: TCP } # mobile speedtest / throughput - - { name: stun, port: 3478, protocol: UDP } # STUN (device<->controller) - - { name: discovery, port: 10001, protocol: UDP } # L2 device discovery + - { name: inform, port: 8080, protocol: TCP } # device inform / adoption + - { name: ui-legacy, port: 8443, protocol: TCP } # legacy Network UI (still served) + - { name: ui, port: 11443, protocol: TCP } # UOS web UI (modern default) + - { name: portal-http, port: 8880, protocol: TCP } # guest portal HTTP + - { name: portal-https, port: 8843, protocol: TCP } # guest portal HTTPS + - { name: throughput, port: 6789, protocol: TCP } # mobile speedtest / throughput + - { name: stun, port: 3478, protocol: UDP } # STUN (device<->controller) + - { name: discovery, port: 10001, protocol: UDP } # L2 device discovery resources: requests: memory: 4Gi diff --git a/apps/vms/unifi/40-networkpolicy.yaml b/apps/vms/unifi/40-networkpolicy.yaml index 73e4fa1..ff297a8 100644 --- a/apps/vms/unifi/40-networkpolicy.yaml +++ b/apps/vms/unifi/40-networkpolicy.yaml @@ -1,5 +1,5 @@ --- -# STAGING / DRAFT — see namespace.yaml header. +# LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header). # # Additive ALLOWs on top of the 10-default-deny baseline. Cilium enforces the UNION of the k8s # NetworkPolicy (default-deny) and these CiliumNetworkPolicies (same pattern as pihole). The VM's @@ -31,13 +31,13 @@ spec: - host toPorts: - ports: - - { port: "8080", protocol: TCP } # inform / adoption + - { port: "8080", protocol: TCP } # inform / adoption - { port: "11443", protocol: TCP } # UOS UI - - { port: "8443", protocol: TCP } # legacy UI - - { port: "8880", protocol: TCP } # guest portal http - - { port: "8843", protocol: TCP } # guest portal https - - { port: "6789", protocol: TCP } # throughput - - { port: "3478", protocol: UDP } # STUN + - { port: "8443", protocol: TCP } # legacy UI + - { port: "8880", protocol: TCP } # guest portal http + - { port: "8843", protocol: TCP } # guest portal https + - { port: "6789", protocol: TCP } # throughput + - { port: "3478", protocol: UDP } # STUN - { port: "10001", protocol: UDP } # discovery --- # 2) EGRESS from the controller: internet (apt, the UOS installer download, UniFi updates + remote diff --git a/apps/vms/unifi/50-service-l4.yaml b/apps/vms/unifi/50-service-l4.yaml index cb01dcf..66d2c56 100644 --- a/apps/vms/unifi/50-service-l4.yaml +++ b/apps/vms/unifi/50-service-l4.yaml @@ -1,5 +1,5 @@ --- -# STAGING / DRAFT — see namespace.yaml header. +# LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header). # # ============================================================================================ # L4 EXPOSURE (the hard part) — why the shared HTTPS Gateway is NOT enough for UniFi. @@ -49,12 +49,12 @@ spec: selector: kubevirt.io/vm: unifi ports: - - { name: inform, port: 8080, targetPort: 8080, protocol: TCP } - - { name: ui, port: 11443, targetPort: 11443, protocol: TCP } - - { name: ui-legacy, port: 8443, targetPort: 8443, protocol: TCP } - - { name: portal-http, port: 8880, targetPort: 8880, protocol: TCP } - - { name: portal-https, port: 8843, targetPort: 8843, protocol: TCP } - - { name: throughput, port: 6789, targetPort: 6789, protocol: TCP } + - { name: inform, port: 8080, targetPort: 8080, protocol: TCP } + - { name: ui, port: 11443, targetPort: 11443, protocol: TCP } + - { name: ui-legacy, port: 8443, targetPort: 8443, protocol: TCP } + - { name: portal-http, port: 8880, targetPort: 8880, protocol: TCP } + - { name: portal-https, port: 8843, targetPort: 8843, protocol: TCP } + - { name: throughput, port: 6789, targetPort: 6789, protocol: TCP } --- apiVersion: v1 kind: Service @@ -72,5 +72,5 @@ spec: selector: kubevirt.io/vm: unifi ports: - - { name: stun, port: 3478, targetPort: 3478, protocol: UDP } + - { name: stun, port: 3478, targetPort: 3478, protocol: UDP } - { name: discovery, port: 10001, targetPort: 10001, protocol: UDP } diff --git a/apps/vms/unifi/README.md b/apps/vms/unifi/README.md index 5a0369a..398e05f 100644 --- a/apps/vms/unifi/README.md +++ b/apps/vms/unifi/README.md @@ -1,8 +1,18 @@ -# UniFi Network controller (UniFi OS Server) — VM onboarding **[STAGING / DRAFT]** +# UniFi Network controller (UniFi OS Server) — VM onboarding -> **NOT WIRED IN.** This directory is deliberately absent from `apps/kustomization.yaml`. -> It lands **after** the storage workstream ships the **`longhorn-vm`** RWX StorageClass; the PM -> then places `unifi` in the correct apps delivery *wave*. Nothing here should be reconciled yet. +> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization +> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate +> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import +> or install cannot stall the app tier. +> +> **Status (verified 2026-08-26):** +> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable). +> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`. +> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`. +> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services. +> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster +> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy). +> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1. ## What this is @@ -18,12 +28,33 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani |---|---| | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation | | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS | -| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (60Gi). First real CDI http import on the cluster | -| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, cloud-init that preps Podman + installs UOS | +| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster | +| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) | | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress | | `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record | -| `90-httproute-optional.yaml` | **Optional, disabled** L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat) | -| `kustomization.yaml` | For `kubectl kustomize apps/unifi` validation only; excludes the optional HTTPRoute | +| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) | + +The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS +serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on +this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands. + +### Cloud-init delivery — two constraints learned the hard way + +1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman + prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the + shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a + Secret deliberately. +2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote + `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml` + therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency. + +### Node prerequisites this VM forced onto the cluster + +- **`nonroot-devices: true`** in the k3s server config (`infra/ansible/roles/k3s_server`) — the + non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without + it: `blockdev: cannot open /dev/cdi-block-volume: Permission denied`. +- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default + 3-replica class made the import fail with insufficient storage. ## UOS install method (researched) @@ -72,38 +103,51 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So: `https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`. -## Open questions / risks (need a PM decision or a live spike) +## Open questions / risks -1. **Unattended install (HIGH).** The installer is a **signed, account-tied** URL that **rotates/ - expires** (cannot be committed) and is documented as **interactive** (no official silent flag). - cloud-init preps the host fully and provides an `install-uos` helper + an optional baked-URL - auto-run that pipes `yes`. **Decision needed:** accept "operator runs `install-uos` at bringup" - vs. invest in a verified silent path. First bringup should be treated as a spike. -2. **CDI http import (MEDIUM).** First http/qcow2 CDI import on this cluster; no checksum on an http - source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress - CNP (included). -3. **Boot firmware (LOW).** Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot. - If not, add the `efi` firmware block (noted inline in `30-virtualmachine.yaml`). -4. **VIP `192.168.1.54` (LOW).** Confirm free, in-pool, and reserved in tofu (Gateway already holds - one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree. -5. **Longhorn RWX prerequisite (BLOCKING).** `longhorn-vm` must exist with RWX/Block + migratable - before apply, else the DataVolume/VM stay Pending. +1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed, + account-tied** URL that **rotates/expires** (cannot be committed) and is documented as + **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for + the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**. + **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or + invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL). +2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI + import on this cluster). Still no checksum on an http source, and the URL tracks a *moving* + Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level + fixes (`nonroot-devices`, `longhorn-scratch`) — see above. +3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS. +4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the + `192.168.1.53–.60` pool (`.53` is the shared Gateway). +5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block / + migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`. 6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access. +7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod + masquerade boundary — see the L4 section. Expect explicit inform-host adoption. -## Post-merge bring-up + verification +## Remaining bring-up -1. **Prereq:** `longhorn-vm` StorageClass live (RWX, Block, migratable). Confirm `192.168.1.54` free. -2. PM adds `apps/unifi` to the right wave in `apps/kustomization.yaml`; reconcile. -3. Watch the CDI import: `kubectl -n unifi get dv unifi-rootdisk -w` → `Succeeded`. -4. VM starts: `kubectl -n unifi get vmi unifi` → `Running`. LB VIP: `kubectl -n unifi get svc` → - both show `192.168.1.54`. -5. **Install UOS** (if not auto-run): get the signed URL from ui.com, then in the guest - `sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos`. Access the guest via - `virtctl console unifi` / SSH once cloud-init adds a key. -6. Verify UI: browse `https://unifi.home.keithjackson.dev:11443`, complete first-run wizard. -7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54`. +Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook. + +1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.** +2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms` + Kustomization (not an app wave). +3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`. +4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show + `192.168.1.54`. +5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key + and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from + ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**. +6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`), + complete the first-run wizard. +7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN. 8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN connects (green). Test guest portal / throughput if used. 9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS. -10. (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version. +10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version. + +**Quick health check today:** +```bash +kubectl -n unifi get dv,vmi,svc +kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}' +``` diff --git a/ci/touched-requirements.sh b/ci/touched-requirements.sh index 2935195..adc4034 100755 --- a/ci/touched-requirements.sh +++ b/ci/touched-requirements.sh @@ -16,9 +16,10 @@ # `conformance.sh --requirements ...` which runs exactly the tests carrying those tags. # # Detection = union of: -# 1. R-* tokens literally present in the diff of changed files — including the -# authoritative `conformance.home.k3s/R-*` tags on changed tests, and the R-IDs -# that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011). +# 1. R-* tokens on the ADDED/REMOVED LINES of the diff, in non-Markdown files — +# including the authoritative `conformance.home.k3s/R-*` tags on changed tests, and +# the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011). +# Untouched lines and prose are NOT scanned; see changed_diff() for why. # 2. A coarse path->requirement fallback for changes that don't self-declare IDs. # The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for # untagged changes so nothing silently escapes the gate. @@ -45,6 +46,36 @@ changed_files() { git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true } +# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding +# documentation. +# +# TWO scoping rules, both learned from the gate blocking merges it should not have: +# +# 1. Scan the DIFF, not whole file contents. Grepping whole files meant a change +# inherited every requirement those files had EVER mentioned, on lines the author +# never touched. Since some of those are legitimately uncovered, `--require-all` +# then failed and BLOCKED the merge over unrelated prose. +# +# 2. Skip `*.md`. A requirement ID in prose is a REFERENCE, not a declaration — +# explaining a requirement is not the same as changing behaviour against it. +# Without this, the decision log cannot record which requirements a change +# concerned without thereby claiming to have touched them, and the CI-fix commit +# that explains a mis-attribution re-triggers the very mis-attribution it fixes. +# Nothing escapes: authoritative declarations (`conformance.home.k3s/R-*` tags, +# Kyverno policies, ServiceDefinitions) all live in YAML, and the doc subtrees +# that matter (conformance/, schemas/, catalog/, clusters/, ci/) are covered by +# the path->requirement fallback regardless of file type. +# +# Removed lines count: deleting a test's requirement tag, or a policy's self-declared +# ID, touches that requirement just as much as adding one. +# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above). Three-dot to +# match changed_files(): changes on this branch since it diverged from base. +changed_diff() { + local base; base="$(resolve_base)" + if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . ':(exclude)*.md' 2>/dev/null; then return; fi + git -C "$REPO_ROOT" diff -U0 "${base}" -- . ':(exclude)*.md' 2>/dev/null || true +} + # Coarse path -> requirement map. Deliberately conservative; extend as subtrees # fill in. Left side is a path prefix (relative to repo root). path_requirements() { @@ -74,12 +105,16 @@ main() { while IFS= read -r f; do [ -n "$f" ] || continue reqs+=" $(path_requirements "$f")" - # Scan file contents for self-declared R-IDs (only if it still exists). - if [ -f "${REPO_ROOT}/${f}" ]; then - reqs+=" $(grep -hoE 'R-[A-Z]+-[0-9]+' "${REPO_ROOT}/${f}" 2>/dev/null | tr '\n' ' ' || true)" - fi done <<< "$files" + # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff). + # `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope. + reqs+=" $(changed_diff \ + | grep -E '^[+-]' \ + | grep -vE '^(\+\+\+|---)' \ + | grep -hoE 'R-[A-Z]+-[0-9]+' \ + | tr '\n' ' ' || true)" + # Normalize: split, unique, sort. Emit to stdout. printf '%s\n' $reqs | sed '/^$/d' | sort -u } diff --git a/conformance/run.sh b/conformance/run.sh index 8a4d84e..277177b 100755 --- a/conformance/run.sh +++ b/conformance/run.sh @@ -60,9 +60,18 @@ FORCE_NO_CLUSTER=0 # path-map keys to it, so editing any ci/ file (SPEC_PIN # adoption included) would false-block. # These are visible GAPs, not hidden — removing them from this list is the right move the -# moment WS-TEST adds their verifying tests. FOLLOW-UP (WS-TEST): the deeper fix is to make -# the path->requirement matcher stop keying on doc-header tokens so real requirements need -# not be exempted; tracked as a harness debt item, not resolved here. +# moment WS-TEST adds their verifying tests. +# +# FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header +# tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in +# ci/touched-requirements.sh: it scans the added/removed lines of the diff, and skips *.md, so +# a requirement named in a header or in prose no longer counts as a declaration. Consequence +# to be aware of: a manifest whose HEADER declares the requirements it serves no longer +# contributes those IDs unless the change actually touches a line carrying them — the +# authoritative signal is the `conformance.home.k3s/R-*` label on a test, which still is. +# The PATH-MAP half remains: ci/touched-requirements.sh keys clusters/* and ci/* to +# R-ENV-02 + R-GITOPS-01 wholesale, which is why those two must stay exempted here. Tighten +# the path map (or scaffold the two requirements) and this list shrinks to R-STATE-02. DOCUMENTED_GAPS="R-STATE-02 R-ENV-02 R-GITOPS-01" usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } diff --git a/infra/PHASE1-SCOPE.md b/infra/PHASE1-SCOPE.md index 8b9a3e1..591d4a9 100644 --- a/infra/PHASE1-SCOPE.md +++ b/infra/PHASE1-SCOPE.md @@ -1,8 +1,22 @@ # Phase 1 Scope — dev/test Cluster Bring-up (local KVM) +> ## ⚠️ HISTORICAL — DO NOT EXECUTE +> +> **Phase 1 is complete.** This document is the *original* scoping for the first dev cluster, which +> ran on **local libvirt/KVM VMs on the workstation** (`10.73.0.0/24`). That cluster was retired: it +> was constrained by the workstation's ~22 GiB RAM and 137 GiB disk, and the dev cluster was migrated +> to a **Proxmox VE** host (nodes `192.168.1.50/.51/.52`, 6 vCPU / 16 GiB each). +> +> Kept for the record — the exit criteria, sizing rationale, and risk analysis below are still the +> reasoning behind the current design. **Everything about libvirt, `virsh`, the `dmacvicar/libvirt` +> provider, `10.73.0.x`, and the "missing prereqs" table is superseded.** +> +> For the cluster as it exists today, read **[`PHASE1-STATUS.md`](PHASE1-STATUS.md)**. + **Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md` (Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux), -0007 (SOPS). **Status:** scoped, awaiting operator go / prerequisite installs. +0007 (SOPS). **Status (as written, 2026-08-16):** scoped, awaiting operator go / prerequisite +installs. **Status now:** delivered, then superseded by the Proxmox migration. ## Objective & exit criteria (roadmap Phase 1) A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC diff --git a/infra/PHASE1-STATUS.md b/infra/PHASE1-STATUS.md index 3711140..1acb930 100644 --- a/infra/PHASE1-STATUS.md +++ b/infra/PHASE1-STATUS.md @@ -1,57 +1,129 @@ -# Phase 1 — Execution Status & Resume Guide +# Dev cluster — current state & reconnect guide -**Checkpoint: 2026-08-16.** The dev/test cluster is **built and running**. Phase 1 is paused before -the Flux/SOPS bootstrap (a deliberate decision — see "Remaining"). This doc is how a future session -picks up. +**Last verified: 2026-08-26.** Phase 1 is **complete and long since exited**; the cluster has been +running the full platform (phases 2–6) for some time. This doc is how a session — human or agent — +reconnects to it and re-establishes the toolchain. -## DONE — a live 3-node HA k3s + Cilium cluster (T-1.1/1.2/1.3) -- `tofu apply` created the `k3s-dev` NAT network (10.73.0.0/24) + 3 VMs (`k3s-dev-1/2/3`, - 6 GiB/4 vCPU, 20 G OS + 40 G data, host-passthrough) — all **running**. -- cloud-init baselined the nodes; k3s HA (embedded etcd, 3 servers) + Cilium installed by Ansible. -- **Live health:** `kubectl get nodes` → **3/3 Ready**; `cilium status` → **OK**; CoreDNS up. +> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation +> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster +> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`, +> or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and +> [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc). -### Fixes applied during execution (committed to main) -- `ansible.cfg`: `stdout_callback yaml` removed in ansible-core 2.20 → builtin default. -- `cilium-values`: `egressGateway` requires `bpf.masquerade: true` (+ `enableIPv4Masquerade`) or the - agent fatals — added. +## The cluster -## Toolchain reality (important for resume) -The pinned **dev container** (`infra/devcontainer/`) holds flux/sops/age/kubectl/helm. BUT: -- **`tofu` runs on the HOST** — the `dmacvicar/libvirt` provider's read-write connect fails inside a - container (`Cannot find start time for pid`). The pinned binary was extracted to - **`~/.local/bin/tofu`** (OpenTofu 1.9.0). Run tofu from the host with `LIBVIRT_DEFAULT_URI=qemu:///system`. -- **`ansible` runs on the HOST** (pre-installed ansible-core 2.20; needs native network to the nodes). - -## State locations (persisted on disk; gitignored) -| What | Path | +| | | |---|---| -| tofu state | `infra/tofu/terraform.tfstate` | -| tofu vars (ssh pubkey) | `infra/tofu/terraform.tfvars` | -| cluster kubeconfig | `infra/ansible/.cache/dev.kubeconfig` (API at 10.73.0.11:6443) | -| SSH key | `~/.ssh/home_network` (user `k3s`, NOPASSWD sudo) | -| VMs / network | libvirt `qemu:///system` (`virsh list`, net `k3s-dev`) | +| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` | +| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 16 GiB, Debian 12 (bookworm) | +| k3s | **v1.31.5+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik | +| CNI | Cilium 1.16.5 (kube-proxy replacement, L2 announcements, egress gateway) | +| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert | +| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` | +| Storage | Longhorn 1.7.3 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 | +| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it | + +Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs, +[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium, +kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`. + +## Network path from the workstation (there is a NAT in the way) + +The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two: + +| Purpose | Workstation-side target | Forwards to | +|---|---|---| +| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` | +| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 | +| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host | + +TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the +VIP is in the serving cert's SAN list, so the NAT address does not have to be. + +There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed. + +## Access + +**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates +against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the +kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`: -## Reconnect / manage -```bash -infra/cluster.sh status # VMs + kubectl get nodes -infra/cluster.sh start|stop # boot/graceful-shutdown the 3 VMs (state persists on disk) -eval "$(infra/cluster.sh kubeconfig)" # export KUBECONFIG -kubectl get nodes -ssh -i ~/.ssh/home_network k3s@10.73.0.11 ``` -The VMs are **running** at checkpoint (~18 GiB RAM). `infra/cluster.sh stop` frees RAM; k3s/etcd/ -Cilium recover on `start` (data on the disks). +kubectl config use-context k3s-oidc +kubectl auth whoami # -> oidc:, groups [oidc:platform-admins ...] +``` -## REMAINING Phase 1 -- **T-1.4 Flux + SOPS bootstrap — Option A (decided):** start the host Gitea (`bootstrap/gitea/`), - push this repo, `flux install`, apply the `flux-system` GitRepository + age Secret — but **keep the - `platform` and `apps` Kustomizations SUSPENDED**. This proves Flux reconciles from the host Gitea - (Phase-1 exit) WITHOUT unleashing the incomplete Phase-0 platform stubs on the working cluster. -- **T-1.5** destroy/rebuild proof (`tofu destroy` a node → `apply` → re-join), then flip - `CLUSTER_AVAILABLE=true` so conformance L2/L3 runs. +`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so +this path needs an interactive desktop. -## Phase 2 prerequisite (before un-suspending `platform/`) -The Phase-0 `platform/` HelmReleases are stubs. **Do NOT let Flux reconcile them onto this cluster -until:** `platform/cilium` `k8sServiceHost` is set to **10.73.0.11** (parity with the Ansible install, -docs/02) and the REPLACE-ME values / secrets (Cloudflare token T-2.4p, etc.) are filled — otherwise -Flux would `helm upgrade` Cilium with a broken value and could break the running CNI. +**kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from +`/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and +add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the +gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need +this, since they cannot complete a browser login. + +**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 k3s@192.168.2.102` (user `k3s`, NOPASSWD sudo). +Ports 2251/2252 for nodes 2/3. + +## Toolchain — use the dev container, not the host + +Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate +toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image: + +```bash +infra/devcontainer/dev.sh build # once +infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs +infra/devcontainer/dev.sh # interactive shell at /work +``` + +Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the +`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket, +which is what broke the old `dmacvicar/libvirt` provider in a container. + +`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring — +**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase +file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability +check runs instead of silently skipping. + +> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, +> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl` +> therefore needs an explicit `DEV_KUBECONFIG=` — the OIDC +> context cannot work there anyway (no browser). Not yet fixed. + +## Off-repo state you must not lose + +| What | Where | +|---|---| +| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret | +| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` | +| Node SSH key | `~/.ssh/home_network` | +| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state | +| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes | +| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` | + +## Health check + +```bash +kubectl get nodes # 3/3 Ready +flux get kustomizations # flux-system, platform, platform-config, + # apps-wave-0/1/2, apps-vms — all Ready +kubectl get helmreleases -A # all Ready +kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db +kubectl get vmi -A # unifi Running +``` + +## Flux's git source is inside the cluster — mind the loop + +`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**: +the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes +through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea, +the Gateway, CoreDNS and the wildcard cert all being up. + +In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without +Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop +only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap +Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that +is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept. + +**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy. +There is no separate "apply" step and no GitHub remote. diff --git a/infra/README.md b/infra/README.md index a0fbc5b..0e729bc 100644 --- a/infra/README.md +++ b/infra/README.md @@ -1,4 +1,29 @@ -# infra/ — Phase 1 node provisioning (local KVM) +# infra/ — node provisioning and the toolchain -OpenTofu (libvirt) + cloud-init + Ansible bring up the 3-node k3s/Cilium cluster. -See PHASE1-SCOPE.md. Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md. +OpenTofu (**Proxmox VE**) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster. +Spec: `../home-network-k3s/docs/02-environments-and-provisioning.md`. + +| Path | What | +|---|---| +| [`tofu/`](tofu/) | The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (`bpg/proxmox` provider) | +| [`cloud-init/`](cloud-init/) | First-boot baseline shipped as a Proxmox user-data snippet | +| [`ansible/`](ansible/) | The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway | +| [`devcontainer/`](devcontainer/) | The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm **and the whole CI gate toolchain**) | +| [`cluster.sh`](cluster.sh) | **RETIRED** — drove the old libvirt VMs; does not work against Proxmox | +| [`PHASE1-STATUS.md`](PHASE1-STATUS.md) | **Current** state of the cluster and how to reconnect to it | +| [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) | Historical Phase-1 scoping doc (libvirt era — superseded) | + +## Where the cluster actually lives + +The dev cluster is **not** on this workstation. It runs on a Proxmox VE host; the nodes sit on the +Proxmox LAN and are reached across a pfSense NAT: + +| | | +|---|---| +| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52` — 6 vCPU, 16 GiB, Debian 12, k3s v1.31.5 | +| kube-API | kube-vip VIP `192.168.1.49:6443`, reached from the workstation via NAT `192.168.2.50:6443` | +| Node SSH | pfSense NAT `192.168.2.102:2250/2251/2252` → node `:22` (Ansible only) | +| Ingress VIP | `192.168.1.53` (Cilium L2, shared Gateway); `.53–.60` reserved for LB VIPs | + +The earlier libvirt/KVM cluster on `10.73.0.0/24` is **retired**. Any doc or command referencing +`virsh`, `qemu:///system`, or `10.73.0.x` for the cluster is historical. diff --git a/infra/cluster.sh b/infra/cluster.sh index ca45394..abfbc84 100755 --- a/infra/cluster.sh +++ b/infra/cluster.sh @@ -1,6 +1,17 @@ #!/usr/bin/env bash -# infra/cluster.sh — manage the local dev k3s cluster (VMs + kubeconfig). Convenience for -# resuming Phase 1 development. Uses host virsh + kubectl (see infra/PHASE1-STATUS.md). +# infra/cluster.sh — RETIRED (libvirt era). DO NOT USE. +# +# This managed the ORIGINAL dev cluster: 3 libvirt/KVM VMs on the workstation, on the +# 10.73.0.0/24 NAT network. That cluster no longer exists — the dev cluster was migrated to +# Proxmox VE (nodes 192.168.1.50/.51/.52) and there are no local `k3s-dev-*` domains for +# virsh to start or stop. It is kept only as a record of the retired lifecycle. +# +# What to use instead (see infra/PHASE1-STATUS.md): +# VM power — the Proxmox UI / API on the Proxmox host, or `tofu` in infra/tofu +# kubeconfig — ~/.kube/config context `k3s-oidc` (Keycloak SSO), or the break-glass +# cert kubeconfig from the primary node +# node access — ssh -p 225{0,1,2} k3s@192.168.2.102 (pfSense NAT) +# cluster health — kubectl get nodes && flux get kustomizations set -euo pipefail export LIBVIRT_DEFAULT_URI=qemu:///system HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 77b5935..189f4df 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -1,51 +1,83 @@ # Dev toolbox (containerized baseline) -The Phase 1 prerequisites are **not all Python** (OpenTofu, `age`, `sops`, `flux` are Go binaries; -only Ansible is Python). Per the operator's rule, the toolchain is therefore a **pinned dev -container** baked with everything needed to drive the local KVM cluster — no host pollution, one -reproducible baseline. +The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`, +`kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's +rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible +baseline, and the *same* binaries CI uses. + +**This is the supported way to run anything in this repo.** Don't install the toolchain on the host. ## What's inside (pinned in `versions.env`) -OpenTofu, `age` + `age-keygen`, `sops`, `flux`, `kubectl`, `helm`, `ansible-core`, `libvirt-client` -(so the tofu `dmacvicar/libvirt` provider can talk to the host libvirt), `qemu-img`, cloud-init seed -tooling (`cloud-utils`/`genisoimage`), `openssh-clients`, and common utilities. + +| Group | Tools | +|---|---| +| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` | +| GitOps / cluster | `flux`, `kubectl`, `helm` | +| Secrets | `sops`, `age` + `age-keygen` | +| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` | +| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) | + +The conformance-tool versions are kept in lockstep with `ci/versions.env` so +`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records +the resolved set at `/etc/dev-toolchain-versions`. ## Use + ```bash cd infra/devcontainer -./dev.sh build # build the image (needs network: base image + pinned binaries) -./dev.sh # build if needed + open a shell in /work (the repo) -./dev.sh run tofu -version +./dev.sh build # build the image (needs network) +./dev.sh # build if needed + shell in /work (the repo) +./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it +./dev.sh run sops -d path/to.sops.yaml +./dev.sh run tofu -chdir=/work/infra/tofu plan ``` `ENGINE=docker ./dev.sh` to use docker instead of rootless podman. -## How it reaches the host KVM -`dev.sh` runs the container with: -- `--network host` — so Ansible can SSH to the cluster VMs on the `k3s-dev` NAT subnet - (10.73.0.0/24) and `kubectl`/`flux` can reach the API server. -- `-v /run/libvirt:/run/libvirt` + `LIBVIRT_DEFAULT_URI=qemu:///system` — so OpenTofu/virsh drive - the host libvirt daemon (the socket is world-writable; no group change needed). -- `--security-opt label=disable` — lets the container use the host libvirt socket without - relabeling it (SELinux hosts). -- `-v $REPO:/work` and `-v ~/.ssh:/root/.ssh:ro` — the repo and your SSH keys. +## What `dev.sh` mounts, and why each one matters -## Known limitation — run `tofu` and `ansible` on the HOST, not in the container -Discovered during Phase 1 bring-up (see `../PHASE1-STATUS.md`): -- **OpenTofu + `dmacvicar/libvirt`** fails *inside* the container — its read-write `qemu:///system` - connect hits a polkit/PID-identity path (`Cannot find start time for pid`) that a mounted socket - can't satisfy across the PID namespace (`--pid=host` didn't help). **Run the pinned tofu on the - host** (extract once: `podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/`; - then `LIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...`). -- **Ansible** talks to the cluster VMs over SSH; simplest to run the pre-installed host `ansible-core` - with native networking to the `k3s-dev` NAT subnet. -- The container remains the baseline for **flux/sops/age/kubectl/helm** (used from inside with - `--network host` once the cluster exists). +- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and + `kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API + (`192.168.2.102:8006`) on the host's network. +- `-v $REPO:/work` (workdir) — the repo. +- `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the + private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with + the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable. +- `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`). +- `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted + `group_vars` decrypt unattended. +- `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when + it exists beside this one. Without it `ci/spec-immutability.sh` honestly *skips* + rather than passing — mount it, or that gate silently verifies nothing. +- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling. + +## tofu and ansible now run **inside** the container + +This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's +`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a +container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and +run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which +talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs +SSH, which `--network host` provides. + +So: run everything in here. The host needs only podman (or docker). + +## kubectl inside the container — known gap + +`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path +that no longer exists**, and does not mount `~/.kube`. Override it explicitly: + +```bash +DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes +``` + +Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser, +which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both +access paths. (Not yet fixed in `dev.sh`.) ## Notes + - Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env` - (one line) to the current stable and rebuild. The image records the resolved set at - `/etc/dev-toolchain-versions`. -- Secrets never enter the image. The SOPS **age private key** is generated at bootstrap and kept - out of git (`.gitignore` guards `age.key`). -- This container is the toolchain, not the cluster. It provisions the 3 KVM nodes (T-1.1–T-1.4); - the cluster itself runs on libvirt on the host. + and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`. +- **Secrets never enter the image** — they arrive only as the bind mounts listed above. +- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see + [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md). diff --git a/infra/tofu/README.md b/infra/tofu/README.md index 436613b..6e5653b 100644 --- a/infra/tofu/README.md +++ b/infra/tofu/README.md @@ -2,12 +2,15 @@ Provisions the "Create" lifecycle stage (docs/02): 3 cloud-init'd VMs on the **Proxmox VE** box, each with an imported OS disk and a blank Longhorn data disk. -**Authoring only — nothing here has been applied.** (Migrated from the old -memory-constrained libvirt host; provider is now `bpg/proxmox`.) +(Migrated from the old memory-constrained libvirt host; provider is now `bpg/proxmox`.) + +**APPLIED AND LIVE.** These three VMs are the running dev cluster — `k3s-dev-1/2/3`, k3s +v1.31.5, carrying the whole platform. Treat any `tofu apply/destroy` here as an operation on +production-for-dev, and note the state file caveat below. Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate). -## What it builds (defaults = VERIFIED Proxmox targets) +## What it builds (defaults = the LIVE cluster, verified 2026-08-26) | Resource | Value | |---|---| @@ -16,11 +19,19 @@ Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate). | Image/snippet datastore | `local` (dir; content = iso + **snippets**) | | Bridge | `vmbr0` (no VLAN) | | Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52`, gw `192.168.1.1`, DNS `192.168.1.18` | -| Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt) | -| OS disk | 40 GiB, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image | -| Data disk | 100 GiB, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init) | +| Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt — real KVM confirmed in-guest) | +| OS disk | **64 GiB**, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image | +| Data disk | **128 GiB**, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init; Longhorn reports ~125 GiB usable per node) | | Cloud-init | native `initialization` (static IP + DNS) + user-data **snippet** for the baseline | -| Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs) | +| Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs). In use: `.53` shared Gateway, `.54` UniFi | + +## ⚠️ State file + +`terraform.tfstate` is gitignored and lives only on the workstation that ran the apply. On a +**fresh workstation it is absent** — and a `tofu apply` with empty state will try to *create* three +VMs that already exist. Recover the state file (or `tofu import` the existing VMs) before running +anything that plans changes. `tofu plan` against empty state is itself harmless, but read the plan: +"3 to add" means you are missing state, not that the cluster is gone. All sizing/IPs/counts/placement are variables — see `variables.tf`. @@ -45,9 +56,11 @@ All sizing/IPs/counts/placement are variables — see `variables.tf`. without `https://`, the provider talks HTTP and eats a 301. Either fix the env file to include the scheme, or export `TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT"` so tofu normalizes it (prepends `https://`). -3. **pfSense SSH NAT** (network-dependent): the PM reaches these nodes and the k3s - API over a pfSense NAT that is not yet created; the Ansible inventory carries a - note to finalize `ansible_host` once the NAT rule exists. +3. **pfSense NAT** — **DONE, live.** The workstation reaches the nodes and the kube-API + across it: SSH `192.168.2.102:2250/2251/2252` → `192.168.1.50/.51/.52:22`, and kube-API + `192.168.2.50:6443` → kube-vip VIP `192.168.1.49:6443`. `infra/ansible/inventory/hosts.ini` + is **hand-maintained** for this mapping — do **not** regenerate it from + `tofu output -raw ansible_inventory`, which knows nothing about the NAT and would overwrite it. ## Auth (env only — never in tofu vars/state) The provider reads these from the environment; the **API token is never a tofu diff --git a/infra/tofu/variables.tf b/infra/tofu/variables.tf index 128aff4..77d7452 100644 --- a/infra/tofu/variables.tf +++ b/infra/tofu/variables.tf @@ -74,7 +74,7 @@ variable "node_count" { } } -# --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 40 GiB OS + 100 GiB data) -- +# --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 64 GiB OS + 128 GiB data) -- variable "vcpu" { description = "vCPUs (cores) per node. Generous — the box has plenty."