jakman85andClaude Opus 4.8
e99e3d1666
Adding the SSH key pushed the inline cloudInitNoCloud.userData over KubeVirt's
2048-byte limit ("userdata exceeds 2048 byte limit. Should use UserDataSecretRef").
Move the cloud-init into apps/vms/unifi/unifi-cloudinit.yaml and wrap it as the
`unifi-cloudinit` Secret via a Kustomize secretGenerator (disableNameSuffixHash
so the VM can reference it by fixed name); VM now uses userDataSecretRef. Content
is non-sensitive (public key + public installer script), so not SOPS-encrypted.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
UniFi Network controller (UniFi OS Server) — VM onboarding [STAGING / DRAFT]
NOT WIRED IN. This directory is deliberately absent from
apps/kustomization.yaml. It lands after the storage workstream ships thelonghorn-vmRWX StorageClass; the PM then placesunifiin the correct apps delivery wave. Nothing here should be reconciled yet.
What this is
A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host
product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer
that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded
datastore inside a Podman container stack — so no separate MongoDB companion is needed
(this supersedes the earlier "unifi needs a Mongo container" note).
Files
| File | Purpose |
|---|---|
namespace.yaml |
unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation |
10-default-deny.yaml |
R-NET-01 baseline: deny all, allow only DNS |
20-datavolume.yaml |
CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (60Gi). First real CDI http import on the cluster |
30-virtualmachine.yaml |
The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, cloud-init that preps Podman + installs UOS |
40-networkpolicy.yaml |
Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
50-service-l4.yaml |
Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record |
90-httproute-optional.yaml |
Optional, disabled L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat) |
kustomization.yaml |
For kubectl kustomize apps/unifi validation only; excludes the optional HTTPRoute |
UOS install method (researched)
- Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+,
slirp4netns1.2+,uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at/usr/bin/podman. - Install: download the signed installer from
fw-download.ubnt.com(URL obtained fromui.com/download/software/unifi-os-server),chmod +x, run as root. UI then onhttps://<host>:11443. - Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.
Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
| Port | Proto | Use |
|---|---|---|
| 11443 | TCP | UOS web UI (modern default) |
| 8443 | TCP | Legacy Network UI (still served) |
| 8080 | TCP | Device inform / adoption |
| 8880 / 8843 | TCP | Guest portal HTTP / HTTPS |
| 6789 | TCP | Mobile throughput/speedtest |
| 3478 | UDP | STUN (device ⇄ controller) |
| 10001 | UDP | L2 device discovery |
L4 exposure design (the hard part)
The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789
or any UDP (3478/10001) — exactly the ports adoption rides. So:
- One VIP, two Services.
unifi-l4-tcp+unifi-l4-udpshare a single LoadBalancer IP via Ciliumlbipam.cilium.io/sharing-key: unifi, pinned to192.168.1.54(lbipam.cilium.io/ips). The VIP is pulled from the existingdefault-pool(192.168.1.53–.60) and ARP-announced on the flat LAN by the existingdefault-l2CiliumL2AnnouncementPolicy — no new Cilium config. selector: kubevirt.io/vm=unifi→ the virt-launcher pod; the VM'smasqueradeiface forwards the declared ports into the guest.externalTrafficPolicy: Localpreserves LAN client source IP.- Device-adoption story: point devices' inform host at the VIP (
http://192.168.1.54:8080/inform) — via the controller's Override inform host setting,set-informon the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway. - Reusable pattern (== the deferred
minecraft:25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.minecraftis the single-TCP-port special case of this multi-port TCP+UDP design.
UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less
https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.
Open questions / risks (need a PM decision or a live spike)
- Unattended install (HIGH). The installer is a signed, account-tied URL that rotates/
expires (cannot be committed) and is documented as interactive (no official silent flag).
cloud-init preps the host fully and provides an
install-uoshelper + an optional baked-URL auto-run that pipesyes. Decision needed: accept "operator runsinstall-uosat bringup" vs. invest in a verified silent path. First bringup should be treated as a spike. - CDI http import (MEDIUM). First http/qcow2 CDI import on this cluster; no checksum on an http source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress CNP (included).
- Boot firmware (LOW). Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot.
If not, add the
efifirmware block (noted inline in30-virtualmachine.yaml). - VIP
192.168.1.54(LOW). Confirm free, in-pool, and reserved in tofu (Gateway already holds one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree. - Longhorn RWX prerequisite (BLOCKING).
longhorn-vmmust exist with RWX/Block + migratable before apply, else the DataVolume/VM stay Pending. - Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.
Post-merge bring-up + verification
- Prereq:
longhorn-vmStorageClass live (RWX, Block, migratable). Confirm192.168.1.54free. - PM adds
apps/unifito the right wave inapps/kustomization.yaml; reconcile. - Watch the CDI import:
kubectl -n unifi get dv unifi-rootdisk -w→Succeeded. - VM starts:
kubectl -n unifi get vmi unifi→Running. LB VIP:kubectl -n unifi get svc→ both show192.168.1.54. - Install UOS (if not auto-run): get the signed URL from ui.com, then in the guest
sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos. Access the guest viavirtctl console unifi/ SSH once cloud-init adds a key. - Verify UI: browse
https://unifi.home.keithjackson.dev:11443, complete first-run wizard. - Verify DNS:
dig unifi.home.keithjackson.dev→192.168.1.54. - Adopt a device: set its inform to
http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used. - Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
- (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.