docs: reconcile the documentation with the deployed reality
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
4cf9572c58
commit
12c40d8c9c
10 files changed
+551
-147
No files matched your search
@@ -922,3 +922,149 @@ R-TLS-01/03 GREEN** (rewrote the probe for LE-prod system-trust + split-horizon
|
||||
LESSON: heavy sustained node load (TCG-emulated VMs) can degrade Cilium datapath -> intermittent LB-VIP
|
||||
reachability; a restart clears it. NOTE: after a host reboot, restart the bootstrap Gitea container
|
||||
(`bash bootstrap/gitea/bootstrap-gitea.sh`) - it is Flux's source and does not auto-start; the k3s VMs auto-recover.
|
||||
|
||||
### BACKFILL — Proxmox migration, app tier, wave split, UniFi VM, kubectl SSO (2026-08-25 → 08-26)
|
||||
**Process note first:** this log had gone stale at `1eb2cb7` (2026-08-25 AM) while `main` advanced
|
||||
**~85 commits** to `3472f1d`. Everything below is reconstructed from those commits and **verified
|
||||
against the live cluster on 2026-08-26**. The lesson is the obvious one: the log is the PM's
|
||||
authoritative record and a two-day gap in it cost a session's worth of re-derivation. Append per
|
||||
work item, not per session.
|
||||
|
||||
**1. INFRA — the dev cluster MIGRATED off libvirt onto Proxmox VE.** The workstation-hosted libvirt
|
||||
cluster (3 VMs on `10.73.0.0/24`, 6 GiB each) was RAM/disk-starved — the KubeVirt work was OOMing
|
||||
nodes (`971c730` had to temp-disable KubeVirt just to relieve pressure). Provisioning was ported
|
||||
`dmacvicar/libvirt` → **`bpg/proxmox`** (`53ca698`), cloud-init moved from a netplan/meta-data seed
|
||||
ISO to a Proxmox **user-data snippet** and then to API-native cloud-init with the rich baseline
|
||||
pushed into the Ansible `node_baseline` role (`39123d7`, `a2392a4`). Nodes are now
|
||||
**`192.168.1.50/.51/.52`, 6 vCPU / 16 GiB / 64 GiB OS + 128 GiB Longhorn data** (`8a8f28b`), Debian
|
||||
12, `cpu_type=host` for nested KVM. Gotchas paid for along the way: the base image must be
|
||||
downloaded as **import** content-type for `import_from` (`fca8d8f`); the Debian *genericcloud*
|
||||
image kernel-panics on first boot under bpg — use the **generic** image + serial console
|
||||
(`28d200c`, upstream bpg#1639); `qemu-guest-agent` must be enable-only when tofu leaves the virtio
|
||||
channel off, or the baseline play fails (`1004c8e`).
|
||||
|
||||
**2. NETWORK — pfSense NAT, then kube-vip, then the tunnel died.** The workstation is not on the
|
||||
cluster LAN. First cut routed Ansible over a pfSense SSH NAT (`192.168.2.102:225x` → node `:22`)
|
||||
and decoupled `k3s_node_ip` from `ansible_host` (`1ff2531`) — `inventory/hosts.ini` is therefore
|
||||
**hand-maintained and must never be regenerated** from `tofu output ansible_inventory`. The
|
||||
kube-API initially needed an SSH tunnel (`0037fb4`). That was retired by **kube-vip**: a
|
||||
control-plane HA VIP at **`192.168.1.49`**, added to the k3s `tls-san` list so TLS validates
|
||||
through the NAT via `tls-server-name` (`60a58f0`, `81ef782`). Controller access is now direct:
|
||||
`https://192.168.2.50:6443` → VIP. Gateway/ingress VIP moved to **`192.168.1.53`** on the Proxmox
|
||||
LAN, and `GATEWAY_VIP` + the split-horizon CoreDNS answer + the DNS conformance tests were all
|
||||
repointed (`604552e`, `63b87ff`).
|
||||
|
||||
**3. COLD-START DEADLOCK, again — and the durable lesson.** Bringing the platform up on the new
|
||||
cluster re-hit the Phase-6 shape: `oauth2-proxy` sits in the `platform` `wait:true` tier but needs
|
||||
Keycloak + trusted TLS + split-horizon DNS, so while it Failed the whole tier stayed not-Ready and
|
||||
`platform-config` (which applies the fix) could never reconcile. Broken the same way as before —
|
||||
temp-disable `oauth2-proxy` and its orphaned `platform-config` refs (`2fea1e2`, `efd6dbe`), let
|
||||
platform go Ready, then re-enable (`e6b3a03`). KubeVirt/CDI re-enabled once the 16 GiB nodes
|
||||
existed (`21669da`). **The FOLLOW-UP from Session 7 is still open and is now twice-proven: move
|
||||
`oauth2-proxy` to a post-platform tier (`dependsOn: platform-config`) so a leaf forward-auth
|
||||
service cannot gate its own dependencies.**
|
||||
|
||||
**4. FLUX SOURCE — now the in-cluster Gitea.** Bootstrap source repointed (`fff4499`), Gitea fixed
|
||||
to mount its PVC at `/data` with a headless-install bootstrap (`73c4475`), then
|
||||
`GitRepository/flux-system` cut over to **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**
|
||||
(`d99d171`). Note the loop this creates: Flux's source is an app Flux deploys, reached via
|
||||
split-horizon DNS → Gateway VIP. Stable in practice (Gitea's Deployment/PVC survive reboots
|
||||
without Flux), but if Gitea is ever *lost* rather than restarted, recovery is the host bootstrap
|
||||
Gitea in `bootstrap/`. `git push origin main` **is** the deploy — there is no other apply step.
|
||||
|
||||
**5. CI — the merge gate is now REAL and in-cluster.** Gitea Actions enabled with an in-cluster
|
||||
`act_runner` + DinD, running the static gate as the single required check on `main` (`bc4519b`).
|
||||
Fixes: kyverno CLI arch is `x86_64` not `amd64`, and the job must wait for DinD certs (`79e7f92`);
|
||||
`make` masked spec-immutability's exit 3 as a 2, so `gate.sh` calls the script directly
|
||||
(`0f75fa2`); `upload-artifact@v4` is unsupported by Gitea Actions (`67bc317`). The **whole test
|
||||
toolchain is now baked into the devcontainer** (`bb48f67`) so `make -C ci gate` on a laptop is
|
||||
byte-identical to CI (R-TEST-02). Kyverno itself needed two real fixes: the `default-deny-shape`
|
||||
policy's scalar-array pattern was **inverted** and silently accepted Ingress-only policies — now
|
||||
expressed via `deny.conditions` requiring both (`1b0848c`); and the fail-closed admission
|
||||
controller was crashlooping into 502s until its resources were raised and dev replicas set to 2
|
||||
(`28a34c4`). Chainsaw parallelism dropped 4→2, then heavy L2/L3 tests serialized, to stop dev etcd
|
||||
overload cascades (`9f52f19`, `948c390`).
|
||||
|
||||
**6. APP TIER — 10 services onboarded through the renderer.** gitea, plex, sonarr, radarr,
|
||||
qbittorrent, jackett (`b76fb7b`…`8392b52`), then seerr (`b6f72ea`), nextcloud (`4be6752`) and
|
||||
semaphore (`77496f9`) — the last two being the **first CNPG-DB + OIDC-target patterns**. Renderer
|
||||
bugs found by real onboarding, each fixed in the renderer rather than by hand-patching output:
|
||||
world/internet egress allow emitted per egress-class (`7e8ae47`); **reciprocal ingress-allow** for
|
||||
declared dependencies, without which a declared dep silently did not connect under default-deny
|
||||
(`14beb85`, R-NET-02); east-west selectors **namespace-pinned** so they match the right pods
|
||||
(`b871877`); `Recreate` strategy for Deployments mounting RWO PVCs, else rollouts deadlock
|
||||
(`d32b5c4`). ServiceDefinition schema gained `env`/`envFrom`/`lifecycle`, retiring the bespoke
|
||||
gitea bootstrap patch (`c8dd4f3`). Semaphore cost two root-causes: `fsGroup` via schema-1.3.0
|
||||
`podSecurityContext` for non-root PVC write (`28498f1`), and **`SEMAPHORE_PORT` being shadowed by
|
||||
kubelet service-link env injection** (`41d696e`) — a trap for any app whose config env name
|
||||
collides with a Service name.
|
||||
|
||||
**7. APP TIER TOPOLOGY — monolithic `apps` split into WAVES.** One `wait:true` Kustomization over
|
||||
all ~130 app resources meant one crashloop blocked every app and every new revision (it bit twice).
|
||||
Split into `apps-wave-0/1/2`, each `dependsOn` the previous, wave assignment computed by
|
||||
`catalog/render.py::compute_waves()` from the declared dependency graph. Design + the data-loss
|
||||
cutover analysis are in **`DESIGN-waves.md`**. The cutover was executed as designed: `prune:false`
|
||||
on the old `apps` first (`1bfb0db`), then the split (`36a1a4e`) so the waves **adopt** by identity
|
||||
instead of delete/recreate. **VERIFIED 2026-08-26:** `nextcloud-db` carries
|
||||
`kustomize.toolkit.fluxcd.io/name=apps-wave-0`, the CNPG Clusters retain their ORIGINAL ages
|
||||
(22h/21h — not recreated), and `prune:true` is restored on the waves. No data was lost.
|
||||
|
||||
**8. PHASE 5 — storage for real VMs, and the UniFi VM.** CSI snapshots + smart-clone + a migratable
|
||||
RWX `longhorn-vm` StorageClass landed to make KubeVirt **live migration** possible (`3586769`);
|
||||
two KubeVirt alerts were plain wrong and firing falsely — a broken http CDI ServiceMonitor causing
|
||||
`CDIOperatorDown` (`6a7e9d1`) and `KubeVirtControlPlaneDown` using a non-existent metric with the
|
||||
wrong job label (`b57b3e9`). **UniFi OS Server** was then onboarded as a VM on a deliberately
|
||||
**non-gating** track (`317b3f0`): `apps/vms/` reconciled by `apps-vms` (`wait:false`,
|
||||
`dependsOn: platform-config`), because a VM's image import + in-guest install is legitimately
|
||||
long-not-Ready and must never gate the app tier. Four cluster-level truths came out of it:
|
||||
- **`nonroot-devices: true`** on the k3s servers — the non-root CDI importer must open a raw Block
|
||||
device k8s never chowns for it (`df5979f`).
|
||||
- **`longhorn-scratch`**, a single-replica class for CDI scratch; the 3-replica default failed the
|
||||
import with insufficient storage (`7a3da96`).
|
||||
- **KubeVirt caps inline `userData` at 2048 bytes.** The rich cloud-init exceeded it; a
|
||||
`userDataSecretRef` was tried and reverted; shipped state is a **minimal ~1.1 KB inline**
|
||||
payload (`e99e3d1`, `760084e`, `ab59202`). The `install-uos` helper did not survive the cut.
|
||||
- **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux envsubst rewrote `${...}` inside
|
||||
the cloud-init and corrupted it (`da1f833`). Do not restore it for consistency with the waves.
|
||||
|
||||
**9. PHASE 6 — kubectl login via Keycloak SSO (`3472f1d`).** A **public** `kubernetes` client with
|
||||
PKCE (no secret in any kubeconfig), a groups mapper, kube-apiserver OIDC flags via the k3s_server
|
||||
role, and `k8s-oidc-rbac.yaml` binding `oidc:platform-admins` → `cluster-admin`,
|
||||
`oidc:viewers` → `view`. The cert kubeconfig is untouched and remains a fully independent
|
||||
**break-glass** path. **VERIFIED LIVE:** `kubectl auth whoami` returns
|
||||
`oidc:<user>` with `[oidc:platform-admins]` — so the API-server flags are active; the rolling
|
||||
restart the commit message deferred has since happened.
|
||||
|
||||
**LIVE STATE VERIFIED 2026-08-26 (fully green):** 3/3 nodes Ready (k3s v1.31.5); **7/7 Flux
|
||||
Kustomizations Ready** at `3472f1d` (flux-system, platform, platform-config, apps-wave-0/1/2,
|
||||
apps-vms); **10/10 HelmReleases Ready**; zero failing pods; 3 CNPG Clusters healthy (keycloak,
|
||||
nextcloud, semaphore); UniFi VMI Running with guest agent connected (Ubuntu 24.04.4,
|
||||
LiveMigratable). Conformance scaffold coverage **36/51 (70.6%)**.
|
||||
|
||||
**⛔ OPEN — `main` is RED on the CI lint gate.** `make -C ci lint` fails with **yamllint `commas`
|
||||
errors in the hand-authored `apps/vms/unifi/` manifests** (`30-virtualmachine.yaml`,
|
||||
`40-networkpolicy.yaml`, `50-service-l4.yaml` — aligned inline comment tables using multiple spaces
|
||||
after a comma). Every other static stage is green in the devcontainer: spec-immutability ✅
|
||||
(spec pinned at `b633b7f`, matching the sibling repo), Kyverno L0 ✅, L1 render 3/3 ✅. This is
|
||||
cosmetic whitespace, but it **blocks the merge gate** and was introduced with the UniFi commits —
|
||||
i.e. the VM work went in without the gate passing. Fix is a whitespace pass; flagged, not yet done.
|
||||
|
||||
**OPEN / NEXT (carried forward):**
|
||||
1. Fix the lint RED above.
|
||||
2. **Install UniFi OS Server in the guest** — the VM runs bare Ubuntu; `:11443` answers nothing
|
||||
(CNP permits `cluster`→`:11443`, so it is genuinely not listening). Install is interactive with
|
||||
a rotating signed URL — needs an operator decision (hand-install vs. a verified silent path).
|
||||
3. **`R-STATE-02`** — the last safety-critical requirement with no L3 test (Longhorn node-loss).
|
||||
Currently exempted via `DOCUMENTED_GAPS`.
|
||||
4. **oauth2-proxy → post-platform tier** (twice-proven cold-start deadlock).
|
||||
5. **OQ-BKP** — off-cluster backup target; gates Velero, CNPG DR depth, and `R-STATE-03`.
|
||||
6. **OQ-2** — the `bulk` StorageClass exists but has `server/share: REPLACE-ME` and provisions
|
||||
nothing; plex/arr/qbittorrent ship config-only until it is resolved.
|
||||
7. **image-automation** still authored-but-disabled (needs a Gitea write-cred).
|
||||
8. **Coverage matcher** — real requirements (`R-ENV-02`, `R-GITOPS-01`) are exempted as
|
||||
DOCUMENTED_GAPS only because the path→requirement matcher is coarse. Fix the matcher (WS-TEST).
|
||||
9. **`dev.sh` KUBECONFIG** defaults to the dead libvirt-era path
|
||||
`infra/ansible/.cache/dev.kubeconfig` and `~/.kube` is not mounted — in-container kubectl needs
|
||||
an explicit `DEV_KUBECONFIG` pointing at a break-glass cert kubeconfig (OIDC needs a browser).
|
||||
10. **tofu state** is absent on a fresh workstation; a `tofu apply` with empty state would try to
|
||||
recreate the three live nodes. Recover the state file or `tofu import` before planning changes.
|
||||
@@ -16,16 +16,63 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
|
||||
|
||||
## Layout
|
||||
```
|
||||
platform/ platform services (Flux HelmReleases/Kustomizations): cilium, longhorn,
|
||||
cert-manager, externaldns, gateway, monitoring, velero, kubevirt,
|
||||
image-automation, keycloak, oauth2-proxy, kyverno/
|
||||
apps/ application workloads, one dir per catalog service
|
||||
catalog/ ServiceDefinition entries (the onboarding surface)
|
||||
schemas/ ServiceDefinition JSON Schema + renderer contract
|
||||
clusters/ Flux entrypoints + dev/ and prod/ overlays
|
||||
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
|
||||
.gitea/ Gitea Actions CI (lint · schema · Kyverno L0 · Chainsaw trigger · merge gate)
|
||||
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
|
||||
pinned dev container that carries the whole toolchain
|
||||
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
|
||||
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
|
||||
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
|
||||
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
|
||||
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
|
||||
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
|
||||
Split out because CRs and the CRDs they use cannot apply in one pass.
|
||||
apps/ application tier, reconciled as dependency-ordered WAVES:
|
||||
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
|
||||
vms/ KubeVirt VirtualMachines (non-gating tier)
|
||||
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
|
||||
schemas/ ServiceDefinition JSON Schema + renderer contract
|
||||
clusters/ Flux entrypoints + dev/ and prod/ overlays
|
||||
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
|
||||
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
|
||||
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
|
||||
```
|
||||
|
||||
Current phase: **Phase 0 — Foundations** (repo, CI, ServiceDefinition schema, conformance harness,
|
||||
Kyverno L0 gate). No cluster exists yet; Phase 1 (cluster bring-up) is not authorized.
|
||||
## Status
|
||||
|
||||
Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
|
||||
open** — `clusters/prod` is a design-only overlay.
|
||||
|
||||
| Phase | Delivered |
|
||||
|---|---|
|
||||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||||
| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
|
||||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
|
||||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||||
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
|
||||
|
||||
**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
|
||||
(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
|
||||
the full report is `conformance/_artifacts/coverage/coverage.md`.
|
||||
|
||||
The running narrative — decisions, root causes, incidents, follow-ups — is
|
||||
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
|
||||
|
||||
## Working on this repo
|
||||
|
||||
Everything the gate needs is baked into the pinned dev container — **do not install the
|
||||
toolchain on the host**:
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh build # once
|
||||
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
|
||||
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
|
||||
```
|
||||
|
||||
The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
|
||||
so the R-TEST-06 spec-immutability check can actually run. See
|
||||
[`infra/devcontainer/README.md`](infra/devcontainer/README.md).
|
||||
|
||||
> **The gate is currently RED on `main`:** yamllint `commas` errors in the hand-authored
|
||||
> `apps/vms/unifi/` manifests. Every other static stage is green. See the last
|
||||
> [`DECISIONS-LOG.md`](DECISIONS-LOG.md) entry.
|
||||
+79
-35
@@ -1,8 +1,18 @@
|
||||
# UniFi Network controller (UniFi OS Server) — VM onboarding **[STAGING / DRAFT]**
|
||||
# UniFi Network controller (UniFi OS Server) — VM onboarding
|
||||
|
||||
> **NOT WIRED IN.** This directory is deliberately absent from `apps/kustomization.yaml`.
|
||||
> It lands **after** the storage workstream ships the **`longhorn-vm`** RWX StorageClass; the PM
|
||||
> then places `unifi` in the correct apps delivery *wave*. Nothing here should be reconciled yet.
|
||||
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
|
||||
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
|
||||
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
|
||||
> or install cannot stall the app tier.
|
||||
>
|
||||
> **Status (verified 2026-08-26):**
|
||||
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
|
||||
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
|
||||
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
|
||||
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
|
||||
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
|
||||
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
|
||||
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
|
||||
|
||||
## What this is
|
||||
|
||||
@@ -18,12 +28,33 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
|
||||
|---|---|
|
||||
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
|
||||
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
|
||||
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (60Gi). First real CDI http import on the cluster |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, cloud-init that preps Podman + installs UOS |
|
||||
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
|
||||
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
|
||||
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
|
||||
| `90-httproute-optional.yaml` | **Optional, disabled** L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat) |
|
||||
| `kustomization.yaml` | For `kubectl kustomize apps/unifi` validation only; excludes the optional HTTPRoute |
|
||||
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
|
||||
|
||||
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
|
||||
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
|
||||
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
|
||||
### Cloud-init delivery — two constraints learned the hard way
|
||||
|
||||
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
|
||||
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
|
||||
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
|
||||
Secret deliberately.
|
||||
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
|
||||
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
|
||||
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
|
||||
|
||||
### Node prerequisites this VM forced onto the cluster
|
||||
|
||||
- **`nonroot-devices: true`** in the k3s server config (`infra/ansible/roles/k3s_server`) — the
|
||||
non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without
|
||||
it: `blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
|
||||
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
|
||||
3-replica class made the import fail with insufficient storage.
|
||||
|
||||
## UOS install method (researched)
|
||||
|
||||
@@ -72,38 +103,51 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
|
||||
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
|
||||
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
|
||||
|
||||
## Open questions / risks (need a PM decision or a live spike)
|
||||
## Open questions / risks
|
||||
|
||||
1. **Unattended install (HIGH).** The installer is a **signed, account-tied** URL that **rotates/
|
||||
expires** (cannot be committed) and is documented as **interactive** (no official silent flag).
|
||||
cloud-init preps the host fully and provides an `install-uos` helper + an optional baked-URL
|
||||
auto-run that pipes `yes`. **Decision needed:** accept "operator runs `install-uos` at bringup"
|
||||
vs. invest in a verified silent path. First bringup should be treated as a spike.
|
||||
2. **CDI http import (MEDIUM).** First http/qcow2 CDI import on this cluster; no checksum on an http
|
||||
source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress
|
||||
CNP (included).
|
||||
3. **Boot firmware (LOW).** Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot.
|
||||
If not, add the `efi` firmware block (noted inline in `30-virtualmachine.yaml`).
|
||||
4. **VIP `192.168.1.54` (LOW).** Confirm free, in-pool, and reserved in tofu (Gateway already holds
|
||||
one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree.
|
||||
5. **Longhorn RWX prerequisite (BLOCKING).** `longhorn-vm` must exist with RWX/Block + migratable
|
||||
before apply, else the DataVolume/VM stay Pending.
|
||||
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
|
||||
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
|
||||
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
|
||||
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
|
||||
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
|
||||
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
|
||||
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
|
||||
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
|
||||
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
|
||||
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
|
||||
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
|
||||
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
|
||||
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
|
||||
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
|
||||
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
|
||||
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
|
||||
installer and for optional remote access.
|
||||
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
|
||||
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
|
||||
|
||||
## Post-merge bring-up + verification
|
||||
## Remaining bring-up
|
||||
|
||||
1. **Prereq:** `longhorn-vm` StorageClass live (RWX, Block, migratable). Confirm `192.168.1.54` free.
|
||||
2. PM adds `apps/unifi` to the right wave in `apps/kustomization.yaml`; reconcile.
|
||||
3. Watch the CDI import: `kubectl -n unifi get dv unifi-rootdisk -w` → `Succeeded`.
|
||||
4. VM starts: `kubectl -n unifi get vmi unifi` → `Running`. LB VIP: `kubectl -n unifi get svc` →
|
||||
both show `192.168.1.54`.
|
||||
5. **Install UOS** (if not auto-run): get the signed URL from ui.com, then in the guest
|
||||
`sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos`. Access the guest via
|
||||
`virtctl console unifi` / SSH once cloud-init adds a key.
|
||||
6. Verify UI: browse `https://unifi.home.keithjackson.dev:11443`, complete first-run wizard.
|
||||
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54`.
|
||||
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
|
||||
|
||||
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
|
||||
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
|
||||
Kustomization (not an app wave).
|
||||
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
|
||||
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
|
||||
`192.168.1.54`.
|
||||
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
|
||||
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
|
||||
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
|
||||
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
|
||||
complete the first-run wizard.
|
||||
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
|
||||
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
|
||||
connects (green). Test guest portal / throughput if used.
|
||||
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
|
||||
10. (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.
|
||||
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
|
||||
|
||||
**Quick health check today:**
|
||||
```bash
|
||||
kubectl -n unifi get dv,vmi,svc
|
||||
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
|
||||
```
|
||||
+15
-1
@@ -1,8 +1,22 @@
|
||||
# Phase 1 Scope — dev/test Cluster Bring-up (local KVM)
|
||||
|
||||
> ## ⚠️ HISTORICAL — DO NOT EXECUTE
|
||||
>
|
||||
> **Phase 1 is complete.** This document is the *original* scoping for the first dev cluster, which
|
||||
> ran on **local libvirt/KVM VMs on the workstation** (`10.73.0.0/24`). That cluster was retired: it
|
||||
> was constrained by the workstation's ~22 GiB RAM and 137 GiB disk, and the dev cluster was migrated
|
||||
> to a **Proxmox VE** host (nodes `192.168.1.50/.51/.52`, 6 vCPU / 16 GiB each).
|
||||
>
|
||||
> Kept for the record — the exit criteria, sizing rationale, and risk analysis below are still the
|
||||
> reasoning behind the current design. **Everything about libvirt, `virsh`, the `dmacvicar/libvirt`
|
||||
> provider, `10.73.0.x`, and the "missing prereqs" table is superseded.**
|
||||
>
|
||||
> For the cluster as it exists today, read **[`PHASE1-STATUS.md`](PHASE1-STATUS.md)**.
|
||||
|
||||
**Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md`
|
||||
(Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux),
|
||||
0007 (SOPS). **Status:** scoped, awaiting operator go / prerequisite installs.
|
||||
0007 (SOPS). **Status (as written, 2026-08-16):** scoped, awaiting operator go / prerequisite
|
||||
installs. **Status now:** delivered, then superseded by the Proxmox migration.
|
||||
|
||||
## Objective & exit criteria (roadmap Phase 1)
|
||||
A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC
|
||||
|
||||
+120
-48
@@ -1,57 +1,129 @@
|
||||
# Phase 1 — Execution Status & Resume Guide
|
||||
# Dev cluster — current state & reconnect guide
|
||||
|
||||
**Checkpoint: 2026-08-16.** The dev/test cluster is **built and running**. Phase 1 is paused before
|
||||
the Flux/SOPS bootstrap (a deliberate decision — see "Remaining"). This doc is how a future session
|
||||
picks up.
|
||||
**Last verified: 2026-08-26.** Phase 1 is **complete and long since exited**; the cluster has been
|
||||
running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
|
||||
reconnects to it and re-establishes the toolchain.
|
||||
|
||||
## DONE — a live 3-node HA k3s + Cilium cluster (T-1.1/1.2/1.3)
|
||||
- `tofu apply` created the `k3s-dev` NAT network (10.73.0.0/24) + 3 VMs (`k3s-dev-1/2/3`,
|
||||
6 GiB/4 vCPU, 20 G OS + 40 G data, host-passthrough) — all **running**.
|
||||
- cloud-init baselined the nodes; k3s HA (embedded etcd, 3 servers) + Cilium installed by Ansible.
|
||||
- **Live health:** `kubectl get nodes` → **3/3 Ready**; `cilium status` → **OK**; CoreDNS up.
|
||||
> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
|
||||
> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
|
||||
> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
|
||||
> or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and
|
||||
> [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc).
|
||||
|
||||
### Fixes applied during execution (committed to main)
|
||||
- `ansible.cfg`: `stdout_callback yaml` removed in ansible-core 2.20 → builtin default.
|
||||
- `cilium-values`: `egressGateway` requires `bpf.masquerade: true` (+ `enableIPv4Masquerade`) or the
|
||||
agent fatals — added.
|
||||
## The cluster
|
||||
|
||||
## Toolchain reality (important for resume)
|
||||
The pinned **dev container** (`infra/devcontainer/`) holds flux/sops/age/kubectl/helm. BUT:
|
||||
- **`tofu` runs on the HOST** — the `dmacvicar/libvirt` provider's read-write connect fails inside a
|
||||
container (`Cannot find start time for pid`). The pinned binary was extracted to
|
||||
**`~/.local/bin/tofu`** (OpenTofu 1.9.0). Run tofu from the host with `LIBVIRT_DEFAULT_URI=qemu:///system`.
|
||||
- **`ansible` runs on the HOST** (pre-installed ansible-core 2.20; needs native network to the nodes).
|
||||
|
||||
## State locations (persisted on disk; gitignored)
|
||||
| What | Path |
|
||||
| | |
|
||||
|---|---|
|
||||
| tofu state | `infra/tofu/terraform.tfstate` |
|
||||
| tofu vars (ssh pubkey) | `infra/tofu/terraform.tfvars` |
|
||||
| cluster kubeconfig | `infra/ansible/.cache/dev.kubeconfig` (API at 10.73.0.11:6443) |
|
||||
| SSH key | `~/.ssh/home_network` (user `k3s`, NOPASSWD sudo) |
|
||||
| VMs / network | libvirt `qemu:///system` (`virsh list`, net `k3s-dev`) |
|
||||
| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
|
||||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 16 GiB, Debian 12 (bookworm) |
|
||||
| k3s | **v1.31.5+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
|
||||
| CNI | Cilium 1.16.5 (kube-proxy replacement, L2 announcements, egress gateway) |
|
||||
| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
|
||||
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
|
||||
| Storage | Longhorn 1.7.3 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
|
||||
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |
|
||||
|
||||
Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
|
||||
[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium,
|
||||
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`.
|
||||
|
||||
## Network path from the workstation (there is a NAT in the way)
|
||||
|
||||
The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two:
|
||||
|
||||
| Purpose | Workstation-side target | Forwards to |
|
||||
|---|---|---|
|
||||
| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` |
|
||||
| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 |
|
||||
| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host |
|
||||
|
||||
TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the
|
||||
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
|
||||
|
||||
There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed.
|
||||
|
||||
## Access
|
||||
|
||||
**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates
|
||||
against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the
|
||||
kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`:
|
||||
|
||||
## Reconnect / manage
|
||||
```bash
|
||||
infra/cluster.sh status # VMs + kubectl get nodes
|
||||
infra/cluster.sh start|stop # boot/graceful-shutdown the 3 VMs (state persists on disk)
|
||||
eval "$(infra/cluster.sh kubeconfig)" # export KUBECONFIG
|
||||
kubectl get nodes
|
||||
ssh -i ~/.ssh/home_network [email protected]
|
||||
```
|
||||
The VMs are **running** at checkpoint (~18 GiB RAM). `infra/cluster.sh stop` frees RAM; k3s/etcd/
|
||||
Cilium recover on `start` (data on the disks).
|
||||
kubectl config use-context k3s-oidc
|
||||
kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
|
||||
```
|
||||
|
||||
## REMAINING Phase 1
|
||||
- **T-1.4 Flux + SOPS bootstrap — Option A (decided):** start the host Gitea (`bootstrap/gitea/`),
|
||||
push this repo, `flux install`, apply the `flux-system` GitRepository + age Secret — but **keep the
|
||||
`platform` and `apps` Kustomizations SUSPENDED**. This proves Flux reconciles from the host Gitea
|
||||
(Phase-1 exit) WITHOUT unleashing the incomplete Phase-0 platform stubs on the working cluster.
|
||||
- **T-1.5** destroy/rebuild proof (`tofu destroy` a node → `apply` → re-join), then flip
|
||||
`CLUSTER_AVAILABLE=true` so conformance L2/L3 runs.
|
||||
`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so
|
||||
this path needs an interactive desktop.
|
||||
|
||||
## Phase 2 prerequisite (before un-suspending `platform/`)
|
||||
The Phase-0 `platform/` HelmReleases are stubs. **Do NOT let Flux reconcile them onto this cluster
|
||||
until:** `platform/cilium` `k8sServiceHost` is set to **10.73.0.11** (parity with the Ansible install,
|
||||
docs/02) and the REPLACE-ME values / secrets (Cloudflare token T-2.4p, etc.) are filled — otherwise
|
||||
Flux would `helm upgrade` Cilium with a broken value and could break the running CNI.
|
||||
**kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from
|
||||
`/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and
|
||||
add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the
|
||||
gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need
|
||||
this, since they cannot complete a browser login.
|
||||
|
||||
**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 [email protected]` (user `k3s`, NOPASSWD sudo).
|
||||
Ports 2251/2252 for nodes 2/3.
|
||||
|
||||
## Toolchain — use the dev container, not the host
|
||||
|
||||
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate
|
||||
toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh build # once
|
||||
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
|
||||
infra/devcontainer/dev.sh # interactive shell at /work
|
||||
```
|
||||
|
||||
Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
|
||||
`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket,
|
||||
which is what broke the old `dmacvicar/libvirt` provider in a container.
|
||||
|
||||
`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring —
|
||||
**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase
|
||||
file, and the sibling spec repo read-only at `/home-network-k3s` so the R-TEST-06 immutability
|
||||
check runs instead of silently skipping.
|
||||
|
||||
> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`,
|
||||
> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl`
|
||||
> therefore needs an explicit `DEV_KUBECONFIG=<path to a break-glass cert kubeconfig>` — the OIDC
|
||||
> context cannot work there anyway (no browser). Not yet fixed.
|
||||
|
||||
## Off-repo state you must not lose
|
||||
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret |
|
||||
| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` |
|
||||
| Node SSH key | `~/.ssh/home_network` |
|
||||
| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state |
|
||||
| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes |
|
||||
| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` |
|
||||
|
||||
## Health check
|
||||
|
||||
```bash
|
||||
kubectl get nodes # 3/3 Ready
|
||||
flux get kustomizations # flux-system, platform, platform-config,
|
||||
# apps-wave-0/1/2, apps-vms — all Ready
|
||||
kubectl get helmreleases -A # all Ready
|
||||
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
|
||||
kubectl get vmi -A # unifi Running
|
||||
```
|
||||
|
||||
## Flux's git source is inside the cluster — mind the loop
|
||||
|
||||
`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**:
|
||||
the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes
|
||||
through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea,
|
||||
the Gateway, CoreDNS and the wildcard cert all being up.
|
||||
|
||||
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
|
||||
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
|
||||
only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap
|
||||
Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that
|
||||
is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept.
|
||||
|
||||
**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy.
|
||||
There is no separate "apply" step and no GitHub remote.
|
||||
+28
-3
@@ -1,4 +1,29 @@
|
||||
# infra/ — Phase 1 node provisioning (local KVM)
|
||||
# infra/ — node provisioning and the toolchain
|
||||
|
||||
OpenTofu (libvirt) + cloud-init + Ansible bring up the 3-node k3s/Cilium cluster.
|
||||
See PHASE1-SCOPE.md. Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
|
||||
OpenTofu (**Proxmox VE**) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
|
||||
Spec: `../home-network-k3s/docs/02-environments-and-provisioning.md`.
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| [`tofu/`](tofu/) | The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (`bpg/proxmox` provider) |
|
||||
| [`cloud-init/`](cloud-init/) | First-boot baseline shipped as a Proxmox user-data snippet |
|
||||
| [`ansible/`](ansible/) | The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
|
||||
| [`devcontainer/`](devcontainer/) | The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm **and the whole CI gate toolchain**) |
|
||||
| [`cluster.sh`](cluster.sh) | **RETIRED** — drove the old libvirt VMs; does not work against Proxmox |
|
||||
| [`PHASE1-STATUS.md`](PHASE1-STATUS.md) | **Current** state of the cluster and how to reconnect to it |
|
||||
| [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) | Historical Phase-1 scoping doc (libvirt era — superseded) |
|
||||
|
||||
## Where the cluster actually lives
|
||||
|
||||
The dev cluster is **not** on this workstation. It runs on a Proxmox VE host; the nodes sit on the
|
||||
Proxmox LAN and are reached across a pfSense NAT:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52` — 6 vCPU, 16 GiB, Debian 12, k3s v1.31.5 |
|
||||
| kube-API | kube-vip VIP `192.168.1.49:6443`, reached from the workstation via NAT `192.168.2.50:6443` |
|
||||
| Node SSH | pfSense NAT `192.168.2.102:2250/2251/2252` → node `:22` (Ansible only) |
|
||||
| Ingress VIP | `192.168.1.53` (Cilium L2, shared Gateway); `.53–.60` reserved for LB VIPs |
|
||||
|
||||
The earlier libvirt/KVM cluster on `10.73.0.0/24` is **retired**. Any doc or command referencing
|
||||
`virsh`, `qemu:///system`, or `10.73.0.x` for the cluster is historical.
|
||||
+13
-2
@@ -1,6 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# infra/cluster.sh — manage the local dev k3s cluster (VMs + kubeconfig). Convenience for
|
||||
# resuming Phase 1 development. Uses host virsh + kubectl (see infra/PHASE1-STATUS.md).
|
||||
# infra/cluster.sh — RETIRED (libvirt era). DO NOT USE.
|
||||
#
|
||||
# This managed the ORIGINAL dev cluster: 3 libvirt/KVM VMs on the workstation, on the
|
||||
# 10.73.0.0/24 NAT network. That cluster no longer exists — the dev cluster was migrated to
|
||||
# Proxmox VE (nodes 192.168.1.50/.51/.52) and there are no local `k3s-dev-*` domains for
|
||||
# virsh to start or stop. It is kept only as a record of the retired lifecycle.
|
||||
#
|
||||
# What to use instead (see infra/PHASE1-STATUS.md):
|
||||
# VM power — the Proxmox UI / API on the Proxmox host, or `tofu` in infra/tofu
|
||||
# kubeconfig — ~/.kube/config context `k3s-oidc` (Keycloak SSO), or the break-glass
|
||||
# cert kubeconfig from the primary node
|
||||
# node access — ssh -p 225{0,1,2} [email protected] (pfSense NAT)
|
||||
# cluster health — kubectl get nodes && flux get kustomizations
|
||||
set -euo pipefail
|
||||
export LIBVIRT_DEFAULT_URI=qemu:///system
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -1,51 +1,83 @@
|
||||
# Dev toolbox (containerized baseline)
|
||||
|
||||
The Phase 1 prerequisites are **not all Python** (OpenTofu, `age`, `sops`, `flux` are Go binaries;
|
||||
only Ansible is Python). Per the operator's rule, the toolchain is therefore a **pinned dev
|
||||
container** baked with everything needed to drive the local KVM cluster — no host pollution, one
|
||||
reproducible baseline.
|
||||
The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`,
|
||||
`kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
|
||||
rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible
|
||||
baseline, and the *same* binaries CI uses.
|
||||
|
||||
**This is the supported way to run anything in this repo.** Don't install the toolchain on the host.
|
||||
|
||||
## What's inside (pinned in `versions.env`)
|
||||
OpenTofu, `age` + `age-keygen`, `sops`, `flux`, `kubectl`, `helm`, `ansible-core`, `libvirt-client`
|
||||
(so the tofu `dmacvicar/libvirt` provider can talk to the host libvirt), `qemu-img`, cloud-init seed
|
||||
tooling (`cloud-utils`/`genisoimage`), `openssh-clients`, and common utilities.
|
||||
|
||||
| Group | Tools |
|
||||
|---|---|
|
||||
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
|
||||
| GitOps / cluster | `flux`, `kubectl`, `helm` |
|
||||
| Secrets | `sops`, `age` + `age-keygen` |
|
||||
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
|
||||
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
|
||||
|
||||
The conformance-tool versions are kept in lockstep with `ci/versions.env` so
|
||||
`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records
|
||||
the resolved set at `/etc/dev-toolchain-versions`.
|
||||
|
||||
## Use
|
||||
|
||||
```bash
|
||||
cd infra/devcontainer
|
||||
./dev.sh build # build the image (needs network: base image + pinned binaries)
|
||||
./dev.sh # build if needed + open a shell in /work (the repo)
|
||||
./dev.sh run tofu -version
|
||||
./dev.sh build # build the image (needs network)
|
||||
./dev.sh # build if needed + shell in /work (the repo)
|
||||
./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
|
||||
./dev.sh run sops -d path/to.sops.yaml
|
||||
./dev.sh run tofu -chdir=/work/infra/tofu plan
|
||||
```
|
||||
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
|
||||
|
||||
## How it reaches the host KVM
|
||||
`dev.sh` runs the container with:
|
||||
- `--network host` — so Ansible can SSH to the cluster VMs on the `k3s-dev` NAT subnet
|
||||
(10.73.0.0/24) and `kubectl`/`flux` can reach the API server.
|
||||
- `-v /run/libvirt:/run/libvirt` + `LIBVIRT_DEFAULT_URI=qemu:///system` — so OpenTofu/virsh drive
|
||||
the host libvirt daemon (the socket is world-writable; no group change needed).
|
||||
- `--security-opt label=disable` — lets the container use the host libvirt socket without
|
||||
relabeling it (SELinux hosts).
|
||||
- `-v $REPO:/work` and `-v ~/.ssh:/root/.ssh:ro` — the repo and your SSH keys.
|
||||
## What `dev.sh` mounts, and why each one matters
|
||||
|
||||
## Known limitation — run `tofu` and `ansible` on the HOST, not in the container
|
||||
Discovered during Phase 1 bring-up (see `../PHASE1-STATUS.md`):
|
||||
- **OpenTofu + `dmacvicar/libvirt`** fails *inside* the container — its read-write `qemu:///system`
|
||||
connect hits a polkit/PID-identity path (`Cannot find start time for pid`) that a mounted socket
|
||||
can't satisfy across the PID namespace (`--pid=host` didn't help). **Run the pinned tofu on the
|
||||
host** (extract once: `podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/`;
|
||||
then `LIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...`).
|
||||
- **Ansible** talks to the cluster VMs over SSH; simplest to run the pre-installed host `ansible-core`
|
||||
with native networking to the `k3s-dev` NAT subnet.
|
||||
- The container remains the baseline for **flux/sops/age/kubectl/helm** (used from inside with
|
||||
`--network host` once the cluster exists).
|
||||
- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
|
||||
`kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API
|
||||
(`192.168.2.102:8006`) on the host's network.
|
||||
- `-v $REPO:/work` (workdir) — the repo.
|
||||
- `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the
|
||||
private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with
|
||||
the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable.
|
||||
- `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`).
|
||||
- `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted
|
||||
`group_vars` decrypt unattended.
|
||||
- `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when
|
||||
it exists beside this one. Without it `ci/spec-immutability.sh` (R-TEST-06) honestly *skips*
|
||||
rather than passing — mount it, or that gate silently verifies nothing.
|
||||
- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling.
|
||||
|
||||
## tofu and ansible now run **inside** the container
|
||||
|
||||
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
|
||||
`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a
|
||||
container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and
|
||||
run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which
|
||||
talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs
|
||||
SSH, which `--network host` provides.
|
||||
|
||||
So: run everything in here. The host needs only podman (or docker).
|
||||
|
||||
## kubectl inside the container — known gap
|
||||
|
||||
`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path
|
||||
that no longer exists**, and does not mount `~/.kube`. Override it explicitly:
|
||||
|
||||
```bash
|
||||
DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
|
||||
```
|
||||
|
||||
Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser,
|
||||
which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both
|
||||
access paths. (Not yet fixed in `dev.sh`.)
|
||||
|
||||
## Notes
|
||||
|
||||
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
|
||||
(one line) to the current stable and rebuild. The image records the resolved set at
|
||||
`/etc/dev-toolchain-versions`.
|
||||
- Secrets never enter the image. The SOPS **age private key** is generated at bootstrap and kept
|
||||
out of git (`.gitignore` guards `age.key`).
|
||||
- This container is the toolchain, not the cluster. It provisions the 3 KVM nodes (T-1.1–T-1.4);
|
||||
the cluster itself runs on libvirt on the host.
|
||||
and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`.
|
||||
- **Secrets never enter the image** — they arrive only as the bind mounts listed above.
|
||||
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
|
||||
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md).
|
||||
+23
-10
@@ -2,12 +2,15 @@
|
||||
|
||||
Provisions the "Create" lifecycle stage (docs/02): 3 cloud-init'd VMs on the
|
||||
**Proxmox VE** box, each with an imported OS disk and a blank Longhorn data disk.
|
||||
**Authoring only — nothing here has been applied.** (Migrated from the old
|
||||
memory-constrained libvirt host; provider is now `bpg/proxmox`.)
|
||||
(Migrated from the old memory-constrained libvirt host; provider is now `bpg/proxmox`.)
|
||||
|
||||
**APPLIED AND LIVE.** These three VMs are the running dev cluster — `k3s-dev-1/2/3`, k3s
|
||||
v1.31.5, carrying the whole platform. Treat any `tofu apply/destroy` here as an operation on
|
||||
production-for-dev, and note the state file caveat below.
|
||||
|
||||
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate).
|
||||
|
||||
## What it builds (defaults = VERIFIED Proxmox targets)
|
||||
## What it builds (defaults = the LIVE cluster, verified 2026-08-26)
|
||||
|
||||
| Resource | Value |
|
||||
|---|---|
|
||||
@@ -16,11 +19,19 @@ Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s substrate).
|
||||
| Image/snippet datastore | `local` (dir; content = iso + **snippets**) |
|
||||
| Bridge | `vmbr0` (no VLAN) |
|
||||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52`, gw `192.168.1.1`, DNS `192.168.1.18` |
|
||||
| Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt) |
|
||||
| OS disk | 40 GiB, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image |
|
||||
| Data disk | 100 GiB, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init) |
|
||||
| Per node | 6 vCPU, 16 GiB RAM, **cpu type `host`** (nested virt for KubeVirt — real KVM confirmed in-guest) |
|
||||
| OS disk | **64 GiB**, virtio0 (`/dev/vda`, boot), imported from the Debian 12 cloud image |
|
||||
| Data disk | **128 GiB**, virtio1 (`/dev/vdb`) → `/var/lib/longhorn` (mounted by cloud-init; Longhorn reports ~125 GiB usable per node) |
|
||||
| Cloud-init | native `initialization` (static IP + DNS) + user-data **snippet** for the baseline |
|
||||
| Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs) |
|
||||
| Reserved | `192.168.1.53–.60` for Cilium L2 ingress VIPs (surfaced in outputs). In use: `.53` shared Gateway, `.54` UniFi |
|
||||
|
||||
## ⚠️ State file
|
||||
|
||||
`terraform.tfstate` is gitignored and lives only on the workstation that ran the apply. On a
|
||||
**fresh workstation it is absent** — and a `tofu apply` with empty state will try to *create* three
|
||||
VMs that already exist. Recover the state file (or `tofu import` the existing VMs) before running
|
||||
anything that plans changes. `tofu plan` against empty state is itself harmless, but read the plan:
|
||||
"3 to add" means you are missing state, not that the cluster is gone.
|
||||
|
||||
All sizing/IPs/counts/placement are variables — see `variables.tf`.
|
||||
|
||||
@@ -45,9 +56,11 @@ All sizing/IPs/counts/placement are variables — see `variables.tf`.
|
||||
without `https://`, the provider talks HTTP and eats a 301. Either fix the env
|
||||
file to include the scheme, or export `TF_VAR_proxmox_endpoint="$PROXMOX_VE_ENDPOINT"`
|
||||
so tofu normalizes it (prepends `https://`).
|
||||
3. **pfSense SSH NAT** (network-dependent): the PM reaches these nodes and the k3s
|
||||
API over a pfSense NAT that is not yet created; the Ansible inventory carries a
|
||||
note to finalize `ansible_host` once the NAT rule exists.
|
||||
3. **pfSense NAT** — **DONE, live.** The workstation reaches the nodes and the kube-API
|
||||
across it: SSH `192.168.2.102:2250/2251/2252` → `192.168.1.50/.51/.52:22`, and kube-API
|
||||
`192.168.2.50:6443` → kube-vip VIP `192.168.1.49:6443`. `infra/ansible/inventory/hosts.ini`
|
||||
is **hand-maintained** for this mapping — do **not** regenerate it from
|
||||
`tofu output -raw ansible_inventory`, which knows nothing about the NAT and would overwrite it.
|
||||
|
||||
## Auth (env only — never in tofu vars/state)
|
||||
The provider reads these from the environment; the **API token is never a tofu
|
||||
|
||||
@@ -74,7 +74,7 @@ variable "node_count" {
|
||||
}
|
||||
}
|
||||
|
||||
# --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 40 GiB OS + 100 GiB data) --
|
||||
# --- per-node sizing (Proxmox target: 6 vCPU / 16 GiB / 64 GiB OS + 128 GiB data) --
|
||||
|
||||
variable "vcpu" {
|
||||
description = "vCPUs (cores) per node. Generous — the box has plenty."
|
||||
|
||||
Reference in new issue
Block a user