Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
7a3da96d9fd4294de406e81b042ad624dbe82bef
home-network-k3s-impl/apps/vms/unifi
T
History
jakman85andClaude Opus 4.8 7a3da96d9f fix(cdi): single-replica scratch class + 40Gi UniFi disk (fix insufficient-storage import) 
The UniFi Block import got past the device-permission fix but then failed
ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk
virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3
(189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node).

- platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass
  (numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is
  waste; if a node dies mid-import CDI just retries.
- platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass ->
  longhorn-scratch.
- apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi;
  40Gi is comfortable for a home site and lighter on the replicated tier).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 15:47:02 -05:00
..
10-default-deny.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
20-datavolume.yaml
fix(cdi): single-replica scratch class + 40Gi UniFi disk (fix insufficient-storage import)
2026-08-26 15:47:02 -05:00
30-virtualmachine.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
40-networkpolicy.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
50-service-l4.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
kustomization.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
namespace.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
README.md
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00

README.md

UniFi Network controller (UniFi OS Server) — VM onboarding [STAGING / DRAFT]

NOT WIRED IN. This directory is deliberately absent from apps/kustomization.yaml. It lands after the storage workstream ships the longhorn-vm RWX StorageClass; the PM then places unifi in the correct apps delivery wave. Nothing here should be reconciled yet.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File Purpose
namespace.yaml unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (60Gi). First real CDI http import on the cluster
30-virtualmachine.yaml The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, cloud-init that preps Podman + installs UOS
40-networkpolicy.yaml Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record
90-httproute-optional.yaml Optional, disabled L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat)
kustomization.yaml For kubectl kustomize apps/unifi validation only; excludes the optional HTTPRoute

UOS install method (researched)

  • Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at /usr/bin/podman.
  • Install: download the signed installer from fw-download.ubnt.com (URL obtained from ui.com/download/software/unifi-os-server), chmod +x, run as root. UI then on https://<host>:11443.
  • Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port Proto Use
11443 TCP UOS web UI (modern default)
8443 TCP Legacy Network UI (still served)
8080 TCP Device inform / adoption
8880 / 8843 TCP Guest portal HTTP / HTTPS
6789 TCP Mobile throughput/speedtest
3478 UDP STUN (device ⇄ controller)
10001 UDP L2 device discovery

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

  • One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned to 192.168.1.54 (lbipam.cilium.io/ips). The VIP is pulled from the existing default-pool (192.168.1.53–.60) and ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config.
  • selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest. externalTrafficPolicy: Local preserves LAN client source IP.
  • Device-adoption story: point devices' inform host at the VIP (http://192.168.1.54:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
  • Reusable pattern (== the deferred minecraft :25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. minecraft is the single-TCP-port special case of this multi-port TCP+UDP design.

UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.

Open questions / risks (need a PM decision or a live spike)

  1. Unattended install (HIGH). The installer is a signed, account-tied URL that rotates/ expires (cannot be committed) and is documented as interactive (no official silent flag). cloud-init preps the host fully and provides an install-uos helper + an optional baked-URL auto-run that pipes yes. Decision needed: accept "operator runs install-uos at bringup" vs. invest in a verified silent path. First bringup should be treated as a spike.
  2. CDI http import (MEDIUM). First http/qcow2 CDI import on this cluster; no checksum on an http source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress CNP (included).
  3. Boot firmware (LOW). Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot. If not, add the efi firmware block (noted inline in 30-virtualmachine.yaml).
  4. VIP 192.168.1.54 (LOW). Confirm free, in-pool, and reserved in tofu (Gateway already holds one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree.
  5. Longhorn RWX prerequisite (BLOCKING). longhorn-vm must exist with RWX/Block + migratable before apply, else the DataVolume/VM stay Pending.
  6. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.

Post-merge bring-up + verification

  1. Prereq: longhorn-vm StorageClass live (RWX, Block, migratable). Confirm 192.168.1.54 free.
  2. PM adds apps/unifi to the right wave in apps/kustomization.yaml; reconcile.
  3. Watch the CDI import: kubectl -n unifi get dv unifi-rootdisk -w → Succeeded.
  4. VM starts: kubectl -n unifi get vmi unifi → Running. LB VIP: kubectl -n unifi get svc → both show 192.168.1.54.
  5. Install UOS (if not auto-run): get the signed URL from ui.com, then in the guest sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos. Access the guest via virtctl console unifi / SSH once cloud-init adds a key.
  6. Verify UI: browse https://unifi.home.keithjackson.dev:11443, complete first-run wizard.
  7. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.1.54.
  8. Adopt a device: set its inform to http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
  9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
  10. (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.
Reference in new issue View Git Blame Copy Permalink