Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs: reconcile the documentation with the deployed reality

Browse Source 
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.

Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.

README.md
  - Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
  - Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
  - Real conformance number (36/51, 70.6%) and how to regenerate it.
  - Devcontainer workflow; note that the gate was red and why.

infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
  cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
  paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
  the toolchain, off-repo state that must not be lost, and the fact that Flux's
  git source is an app inside the cluster (`git push origin main` IS the deploy).

infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
  rationale are still worth keeping; the libvirt execution steps are not.

infra/README.md — Proxmox, plus a map of the subtree and where the cluster
  actually lives (it is not on this workstation).

infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
  is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
  talks to an HTTP API. Everything runs in the container now. Documents the full
  CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
  and the read-only spec repo especially — without the latter, spec-immutability
  silently skips instead of verifying).

infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
  is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
  against the running nodes), pfSense NAT marked done, and a warning that
  terraform.tfstate is absent on a fresh workstation — an apply with empty state
  would try to recreate the three live nodes.

infra/tofu/variables.tf — section comment disagreed with its own defaults.

infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
  domains that do not exist; behaviour unchanged, but it no longer lies.

apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
  since been dropped. Now: live status, 40Gi, resolved open questions marked
  resolved, and the two constraints the next person will otherwise rediscover —
  KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
  postBuild.substituteFrom (envsubst corrupts the cloud-init).

DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
  migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
  the Flux source cutover, CI going live in-cluster, ten app onboardings and the
  renderer bugs they exposed, the wave split (verified: DBs adopted, not
  recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
  VM, and kubectl SSO. Plus verified live state and the carried-forward open list.

Two corrections to the record while verifying:
  - kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
    [oidc:platform-admins]), so the rolling restart 3472f1d deferred has
    happened.
  - UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
    24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
    and the CNP explicitly permits cluster -> :11443, so it is genuinely not
    listening rather than a policy artifact.

Docs and comments only; no manifest semantics change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-08-27 00:00:05 -05:00
1 parent 4cf9572c58
commit 12c40d8c9c
10 files changed
+551 -147
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/vms/unifi/README.md
+79 -35
View File
@@ -1,8 +1,18 @@
# UniFi Network controller (UniFi OS Server) — VM onboarding **[STAGING / DRAFT]**
# UniFi Network controller (UniFi OS Server) — VM onboarding
> **NOT WIRED IN.** This directory is deliberately absent from `apps/kustomization.yaml`.
> It lands **after** the storage workstream ships the **`longhorn-vm`** RWX StorageClass; the PM
> then places `unifi` in the correct apps delivery *wave*. Nothing here should be reconciled yet.
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
> or install cannot stall the app tier.
>
> **Status (verified 2026-08-26):**
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
## What this is
@@ -18,12 +28,33 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
|---|---|
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (60Gi). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, cloud-init that preps Podman + installs UOS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
| `90-httproute-optional.yaml` | **Optional, disabled** L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat) |
| `kustomization.yaml` | For `kubectl kustomize apps/unifi` validation only; excludes the optional HTTPRoute |
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
### Cloud-init delivery — two constraints learned the hard way
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
Secret deliberately.
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
### Node prerequisites this VM forced onto the cluster
- **`nonroot-devices: true`** in the k3s server config (`infra/ansible/roles/k3s_server`) — the
non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without
it: `blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
3-replica class made the import fail with insufficient storage.
## UOS install method (researched)
@@ -72,38 +103,51 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
## Open questions / risks (need a PM decision or a live spike)
## Open questions / risks
1. **Unattended install (HIGH).** The installer is a **signed, account-tied** URL that **rotates/
expires** (cannot be committed) and is documented as **interactive** (no official silent flag).
cloud-init preps the host fully and provides an `install-uos` helper + an optional baked-URL
auto-run that pipes `yes`. **Decision needed:** accept "operator runs `install-uos` at bringup"
vs. invest in a verified silent path. First bringup should be treated as a spike.
2. **CDI http import (MEDIUM).** First http/qcow2 CDI import on this cluster; no checksum on an http
source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress
CNP (included).
3. **Boot firmware (LOW).** Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot.
If not, add the `efi` firmware block (noted inline in `30-virtualmachine.yaml`).
4. **VIP `192.168.1.54` (LOW).** Confirm free, in-pool, and reserved in tofu (Gateway already holds
one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree.
5. **Longhorn RWX prerequisite (BLOCKING).** `longhorn-vm` must exist with RWX/Block + migratable
before apply, else the DataVolume/VM stay Pending.
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
installer and for optional remote access.
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
## Post-merge bring-up + verification
## Remaining bring-up
1. **Prereq:** `longhorn-vm` StorageClass live (RWX, Block, migratable). Confirm `192.168.1.54` free.
2. PM adds `apps/unifi` to the right wave in `apps/kustomization.yaml`; reconcile.
3. Watch the CDI import: `kubectl -n unifi get dv unifi-rootdisk -w` → `Succeeded`.
4. VM starts: `kubectl -n unifi get vmi unifi` → `Running`. LB VIP: `kubectl -n unifi get svc` →
both show `192.168.1.54`.
5. **Install UOS** (if not auto-run): get the signed URL from ui.com, then in the guest
`sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos`. Access the guest via
`virtctl console unifi` / SSH once cloud-init adds a key.
6. Verify UI: browse `https://unifi.home.keithjackson.dev:11443`, complete first-run wizard.
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54`.
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
Kustomization (not an app wave).
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
`192.168.1.54`.
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
complete the first-run wizard.
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
connects (green). Test guest portal / throughput if used.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
**Quick health check today:**
```bash
kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
```