Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track

Browse Source 
First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is
Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi
OS + Network app + its own datastore, so no MongoDB companion.

- apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm
  RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate)
  with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer
  NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public
  firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned
  URL to maintain (currently UOS v5.1.37).
- L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced
  Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7
  Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443.
- default-deny + additive CNPs (controller ingress, egress, and CDI-importer
  egress for the image pull).
- clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false,
  dependsOn platform-config) — VMs are a separate lifecycle from the app waves,
  so a slow VM import/install can't stall the app tier.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-26 15:05:05 -05:00
1 parent 36a1a4e1f9
commit 317b3f02f1
11 files changed
+561
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/vms/unifi/README.md
+109
View File
@@ -0,0 +1,109 @@
# UniFi Network controller (UniFi OS Server) — VM onboarding **[STAGING / DRAFT]**
> **NOT WIRED IN.** This directory is deliberately absent from `apps/kustomization.yaml`.
> It lands **after** the storage workstream ships the **`longhorn-vm`** RWX StorageClass; the PM
> then places `unifi` in the correct apps delivery *wave*. Nothing here should be reconciled yet.
## What this is
A KubeVirt `VirtualMachine` running **UniFi OS Server (UOS)** — Ubiquiti's current self-host
product. The legacy `unifi` `.deb`/`unifi-network-server` is deprecated. UOS is a single installer
that runs on Ubuntu Server and bundles **UniFi OS + the Network application + its own embedded
datastore** inside a Podman container stack — so **no separate MongoDB companion is needed**
(this supersedes the earlier "unifi needs a Mongo container" note).
## Files
| File | Purpose |
|---|---|
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (60Gi). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, cloud-init that preps Podman + installs UOS |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
| `90-httproute-optional.yaml` | **Optional, disabled** L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat) |
| `kustomization.yaml` | For `kubectl kustomize apps/unifi` validation only; excludes the optional HTTPRoute |
## UOS install method (researched)
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
`ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
**`https://<host>:11443`**.
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
| Port | Proto | Use |
|---|---|---|
| 11443 | TCP | UOS web UI (modern default) |
| 8443 | TCP | Legacy Network UI (still served) |
| 8080 | TCP | **Device inform / adoption** |
| 8880 / 8843 | TCP | Guest portal HTTP / HTTPS |
| 6789 | TCP | Mobile throughput/speedtest |
| 3478 | UDP | **STUN** (device ⇄ controller) |
| 10001 | UDP | **L2 device discovery** |
## L4 exposure design (the hard part)
The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
— via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
`minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
## Open questions / risks (need a PM decision or a live spike)
1. **Unattended install (HIGH).** The installer is a **signed, account-tied** URL that **rotates/
expires** (cannot be committed) and is documented as **interactive** (no official silent flag).
cloud-init preps the host fully and provides an `install-uos` helper + an optional baked-URL
auto-run that pipes `yes`. **Decision needed:** accept "operator runs `install-uos` at bringup"
vs. invest in a verified silent path. First bringup should be treated as a spike.
2. **CDI http import (MEDIUM).** First http/qcow2 CDI import on this cluster; no checksum on an http
source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress
CNP (included).
3. **Boot firmware (LOW).** Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot.
If not, add the `efi` firmware block (noted inline in `30-virtualmachine.yaml`).
4. **VIP `192.168.1.54` (LOW).** Confirm free, in-pool, and reserved in tofu (Gateway already holds
one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree.
5. **Longhorn RWX prerequisite (BLOCKING).** `longhorn-vm` must exist with RWX/Block + migratable
before apply, else the DataVolume/VM stay Pending.
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
installer and for optional remote access.
## Post-merge bring-up + verification
1. **Prereq:** `longhorn-vm` StorageClass live (RWX, Block, migratable). Confirm `192.168.1.54` free.
2. PM adds `apps/unifi` to the right wave in `apps/kustomization.yaml`; reconcile.
3. Watch the CDI import: `kubectl -n unifi get dv unifi-rootdisk -w` → `Succeeded`.
4. VM starts: `kubectl -n unifi get vmi unifi` → `Running`. LB VIP: `kubectl -n unifi get svc` →
both show `192.168.1.54`.
5. **Install UOS** (if not auto-run): get the signed URL from ui.com, then in the guest
`sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos`. Access the guest via
`virtctl console unifi` / SSH once cloud-init adds a key.
6. Verify UI: browse `https://unifi.home.keithjackson.dev:11443`, complete first-run wizard.
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54`.
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
connects (green). Test guest portal / throughput if used.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.