Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
330c99eb8bddb9ca5ff3a38adf32fc1153fb7ce5
home-network-k3s-impl/apps/vms/unifi/README.md
T
jakman85andClaude Opus 5 12c40d8c9c docs: reconcile the documentation with the deployed reality 
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.

Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.

README.md
  - Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
  - Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
  - Real conformance number (36/51, 70.6%) and how to regenerate it.
  - Devcontainer workflow; note that the gate was red and why.

infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
  cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
  paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
  the toolchain, off-repo state that must not be lost, and the fact that Flux's
  git source is an app inside the cluster (`git push origin main` IS the deploy).

infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
  rationale are still worth keeping; the libvirt execution steps are not.

infra/README.md — Proxmox, plus a map of the subtree and where the cluster
  actually lives (it is not on this workstation).

infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
  is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
  talks to an HTTP API. Everything runs in the container now. Documents the full
  CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
  and the read-only spec repo especially — without the latter, spec-immutability
  silently skips instead of verifying).

infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
  is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
  against the running nodes), pfSense NAT marked done, and a warning that
  terraform.tfstate is absent on a fresh workstation — an apply with empty state
  would try to recreate the three live nodes.

infra/tofu/variables.tf — section comment disagreed with its own defaults.

infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
  domains that do not exist; behaviour unchanged, but it no longer lies.

apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
  since been dropped. Now: live status, 40Gi, resolved open questions marked
  resolved, and the two constraints the next person will otherwise rediscover —
  KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
  postBuild.substituteFrom (envsubst corrupts the cloud-init).

DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
  migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
  the Flux source cutover, CI going live in-cluster, ten app onboardings and the
  renderer bugs they exposed, the wave split (verified: DBs adopted, not
  recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
  VM, and kubectl SSO. Plus verified live state and the carried-forward open list.

Two corrections to the record while verifying:
  - kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
    [oidc:platform-admins]), so the rolling restart 3472f1d deferred has
    happened.
  - UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
    24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
    and the CNP explicitly permits cluster -> :11443, so it is genuinely not
    listening rather than a policy artifact.

Docs and comments only; no manifest semantics change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-27 00:00:05 -05:00

154 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# UniFi Network controller (UniFi OS Server) — VM onboarding
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
> or install cannot stall the app tier.
>
> **Status (verified 2026-08-26):**
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
## What this is
A KubeVirt `VirtualMachine` running **UniFi OS Server (UOS)** — Ubiquiti's current self-host
product. The legacy `unifi` `.deb`/`unifi-network-server` is deprecated. UOS is a single installer
that runs on Ubuntu Server and bundles **UniFi OS + the Network application + its own embedded
datastore** inside a Podman container stack — so **no separate MongoDB companion is needed**
(this supersedes the earlier "unifi needs a Mongo container" note).
## Files
| File | Purpose |
|---|---|
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
### Cloud-init delivery — two constraints learned the hard way
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
Secret deliberately.
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
### Node prerequisites this VM forced onto the cluster
- **`nonroot-devices: true`** in the k3s server config (`infra/ansible/roles/k3s_server`) — the
non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without
it: `blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
3-replica class made the import fail with insufficient storage.
## UOS install method (researched)
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
`ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
**`https://<host>:11443`**.
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
| Port | Proto | Use |
|---|---|---|
| 11443 | TCP | UOS web UI (modern default) |
| 8443 | TCP | Legacy Network UI (still served) |
| 8080 | TCP | **Device inform / adoption** |
| 8880 / 8843 | TCP | Guest portal HTTP / HTTPS |
| 6789 | TCP | Mobile throughput/speedtest |
| 3478 | UDP | **STUN** (device ⇄ controller) |
| 10001 | UDP | **L2 device discovery** |
## L4 exposure design (the hard part)
The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
— via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
`minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
## Open questions / risks
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
installer and for optional remote access.
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
## Remaining bring-up
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
Kustomization (not an app wave).
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
`192.168.1.54`.
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
complete the first-run wizard.
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
connects (green). Test guest portal / throughput if used.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
**Quick health check today:**
```bash
kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
```
Reference in new issue View Git Blame Copy Permalink