home-network-k3s-impl/apps/vms/unifi/README.md

UniFi Network controller (UniFi OS Server) — VM onboarding

LIVE — partially. Reconciled by the non-gating apps-vms Flux Kustomization (clusters/base/apps-vms.yaml, dependsOn: platform-config, wait: false). VMs are a separate lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install cannot stall the app tier.

Status (verified 2026-08-26):

• longhorn-vm StorageClass — live (RWX / Block / migratable).
• CDI import — Succeeded (100%); unifi-rootdisk PVC Bound, 40Gi, RWX, longhorn-vm.
• VM — Running, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on k3s-dev-1.
• L4 LoadBalancer VIP 192.168.1.54 — allocated on both Services.
• UniFi OS Server itself is NOT installed. Port 11443 does not answer from in-cluster (the CNP explicitly permits cluster → :11443, so this is "nothing listening", not policy). The guest is a bare, prepped Ubuntu. This is the one remaining step — see Open question 1.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File	Purpose
namespace.yaml	unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml	R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml	CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (40Gi). First real CDI http import on the cluster
30-virtualmachine.yaml	The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, minimal inline cloud-init (SSH key, qemu-guest-agent, Podman prep)
40-networkpolicy.yaml	Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml	Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record
kustomization.yaml	The reconciled resource list (referenced by apps/vms/kustomization.yaml)

The optional L7 HTTPRoute (90-httproute-optional.yaml) was dropped, not merely disabled: UOS serves HTTPS on :11443, so a Gateway route needs a BackendTLSPolicy that is not confirmed on this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.

Cloud-init delivery — two constraints learned the hard way

1. KubeVirt caps inline userData at 2048 bytes. The original rich cloud-init (full Podman prep + install-uos helper) blew past it. A userDataSecretRef was tried, then reverted; the shipped config is a minimal ~1.1 KB inline userData. Keep additions small or move to a Secret deliberately.
2. apps-vms must NOT use postBuild.substituteFrom. Flux variable substitution rewrote $-bearing text inside the cloud-init payload and corrupted it. clusters/base/apps-vms.yaml therefore carries no substituteFrom — unlike every app wave. Do not "restore" it for consistency.

Node prerequisites this VM forced onto the cluster

• nonroot-devices: true in the k3s server config (infra/ansible/roles/k3s_server) — the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without it: blockdev: cannot open /dev/cdi-block-volume: Permission denied.
• longhorn-scratch — a single-replica StorageClass for CDI scratch space; the default 3-replica class made the import fail with insufficient storage.

UOS install method (researched)

• Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at /usr/bin/podman.
• Install: download the signed installer from fw-download.ubnt.com (URL obtained from ui.com/download/software/unifi-os-server), chmod +x, run as root. UI then on https://<host>:11443.
• Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port	Proto	Use
11443	TCP	UOS web UI (modern default)
8443	TCP	Legacy Network UI (still served)
8080	TCP	Device inform / adoption
8880 / 8843	TCP	Guest portal HTTP / HTTPS
6789	TCP	Mobile throughput/speedtest
3478	UDP	STUN (device ⇄ controller)
10001	UDP	L2 device discovery

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

• One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned to 192.168.1.54 (lbipam.cilium.io/ips). The VIP is pulled from the existing default-pool (192.168.1.53–.60) and ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config.
• selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest. externalTrafficPolicy: Local preserves LAN client source IP.
• Device-adoption story: point devices' inform host at the VIP (http://192.168.1.54:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
• Reusable pattern (== the deferred minecraft :25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. minecraft is the single-TCP-port special case of this multi-port TCP+UDP design.

UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.

Open questions / risks

1. Unattended install (HIGH) — STILL OPEN. This is the blocker. The installer is a signed, account-tied URL that rotates/expires (cannot be committed) and is documented as interactive (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for the inline userData limit, so the install-uos helper is no longer baked into the guest. Decision needed: operator runs the install by hand at bring-up (current de-facto answer), or invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
2. CDI http import (MEDIUM) — RESOLVED in practice. The import succeeded (first http/qcow2 CDI import on this cluster). Still no checksum on an http source, and the URL tracks a moving Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level fixes (nonroot-devices, longhorn-scratch) — see above.
3. Boot firmware (LOW) — RESOLVED. Boots as shipped; running Ubuntu 24.04.4 LTS.
4. VIP 192.168.1.54 (LOW) — RESOLVED. Allocated and held by both Services, inside the 192.168.1.53–.60 pool (.53 is the shared Gateway).
5. Longhorn RWX prerequisite (was BLOCKING) — RESOLVED. longhorn-vm is live (RWX / Block / migratable); the VMI reports LiveMigratable=True and StorageLiveMigratable=True.
6. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.
7. Broadcast adoption (LOW, by design). L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary — see the L4 section. Expect explicit inform-host adoption.

Remaining bring-up

Steps 1–4 below are done. The sequence is kept because it is also the recovery/rebuild runbook.

1. longhorn-vm StorageClass live; 192.168.1.54 free. Done.
2. Wire into a reconciled path — done, via apps/vms/ + the non-gating apps-vms Kustomization (not an app wave).
3. CDI import — done: kubectl -n unifi get dv unifi-rootdisk → Succeeded.
4. VM + VIP — done: kubectl -n unifi get vmi unifi → Running; both Services show 192.168.1.54.
5. → NEXT: install UOS in the guest. SSH in (the inline cloud-init installs the operator's key and qemu-guest-agent) or virtctl console unifi, fetch the signed installer URL from ui.com/download/software/unifi-os-server, then chmod +x and run it as root.
6. Verify the UI: https://unifi.home.keithjackson.dev:11443 (or https://192.168.1.54:11443), complete the first-run wizard.
7. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.1.54 on the cluster LAN.
8. Adopt a device: set its inform to http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Re-add the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.

Quick health check today:

kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'