Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
df5979fceb99dbe174982f3bf84656f6a8d1ff77
home-network-k3s-impl/apps/vms/unifi
T
History
jakman85andClaude Opus 4.8 df5979fceb fix(k3s): enable nonroot-devices so CDI can import to Block-mode PVCs 
The UniFi VM boot-disk DataVolume (RWX + Block on longhorn-vm) failed to import:
CDI's non-root importer (UID 107, cap-drop ALL) hit
`blockdev: cannot open /dev/cdi-block-volume: Permission denied`.

Kubernetes only applies fsGroup ownership to *filesystem* volumes — it never
chowns a raw block device — and Longhorn's CSIDriver (fsGroupPolicy:
ReadWriteOnceWithFSType) skips fsGroup on RWX/Block volumes regardless, so the
device stays root:root. The CDI-documented fix is the container-runtime setting
`device_ownership_from_security_context`, exposed on k3s (>= v1.31.3+k3s1, this
cluster is v1.31.5) as `nonroot-devices: true`. It makes containerd set the
in-container device node's owner from the pod's securityContext.

Added to the Ansible-managed /etc/rancher/k3s/config.yaml alongside tls-san;
applied via the existing rolling `Restart k3s` handler (serial:1, kube-vip keeps
the API up, etcd holds 2/3). Node-scoped, security-neutral (pods stay non-root),
and preserves the longhorn-vm RWX/Block/migratable contract unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 15:24:07 -05:00
..
10-default-deny.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
20-datavolume.yaml
fix(k3s): enable nonroot-devices so CDI can import to Block-mode PVCs
2026-08-26 15:24:07 -05:00
30-virtualmachine.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
40-networkpolicy.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
50-service-l4.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
kustomization.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
namespace.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
README.md
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00

README.md

UniFi Network controller (UniFi OS Server) — VM onboarding [STAGING / DRAFT]

NOT WIRED IN. This directory is deliberately absent from apps/kustomization.yaml. It lands after the storage workstream ships the longhorn-vm RWX StorageClass; the PM then places unifi in the correct apps delivery wave. Nothing here should be reconciled yet.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File Purpose
namespace.yaml unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (60Gi). First real CDI http import on the cluster
30-virtualmachine.yaml The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, cloud-init that preps Podman + installs UOS
40-networkpolicy.yaml Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record
90-httproute-optional.yaml Optional, disabled L7 UI route via the shared Gateway (needs BackendTLSPolicy — see caveat)
kustomization.yaml For kubectl kustomize apps/unifi validation only; excludes the optional HTTPRoute

UOS install method (researched)

  • Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at /usr/bin/podman.
  • Install: download the signed installer from fw-download.ubnt.com (URL obtained from ui.com/download/software/unifi-os-server), chmod +x, run as root. UI then on https://<host>:11443.
  • Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port Proto Use
11443 TCP UOS web UI (modern default)
8443 TCP Legacy Network UI (still served)
8080 TCP Device inform / adoption
8880 / 8843 TCP Guest portal HTTP / HTTPS
6789 TCP Mobile throughput/speedtest
3478 UDP STUN (device ⇄ controller)
10001 UDP L2 device discovery

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

  • One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned to 192.168.1.54 (lbipam.cilium.io/ips). The VIP is pulled from the existing default-pool (192.168.1.53–.60) and ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config.
  • selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest. externalTrafficPolicy: Local preserves LAN client source IP.
  • Device-adoption story: point devices' inform host at the VIP (http://192.168.1.54:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
  • Reusable pattern (== the deferred minecraft :25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. minecraft is the single-TCP-port special case of this multi-port TCP+UDP design.

UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.

Open questions / risks (need a PM decision or a live spike)

  1. Unattended install (HIGH). The installer is a signed, account-tied URL that rotates/ expires (cannot be committed) and is documented as interactive (no official silent flag). cloud-init preps the host fully and provides an install-uos helper + an optional baked-URL auto-run that pipes yes. Decision needed: accept "operator runs install-uos at bringup" vs. invest in a verified silent path. First bringup should be treated as a spike.
  2. CDI http import (MEDIUM). First http/qcow2 CDI import on this cluster; no checksum on an http source. Consider pinning a dated build dir for reproducibility. Import needs the importer-egress CNP (included).
  3. Boot firmware (LOW). Ubuntu cloud images are hybrid BIOS+UEFI; default SeaBIOS should boot. If not, add the efi firmware block (noted inline in 30-virtualmachine.yaml).
  4. VIP 192.168.1.54 (LOW). Confirm free, in-pool, and reserved in tofu (Gateway already holds one pool IP, typically .53). Could not verify live — no kubeconfig cache in this worktree.
  5. Longhorn RWX prerequisite (BLOCKING). longhorn-vm must exist with RWX/Block + migratable before apply, else the DataVolume/VM stay Pending.
  6. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.

Post-merge bring-up + verification

  1. Prereq: longhorn-vm StorageClass live (RWX, Block, migratable). Confirm 192.168.1.54 free.
  2. PM adds apps/unifi to the right wave in apps/kustomization.yaml; reconcile.
  3. Watch the CDI import: kubectl -n unifi get dv unifi-rootdisk -w → Succeeded.
  4. VM starts: kubectl -n unifi get vmi unifi → Running. LB VIP: kubectl -n unifi get svc → both show 192.168.1.54.
  5. Install UOS (if not auto-run): get the signed URL from ui.com, then in the guest sudo UOS_URL='https://fw-download.ubnt.com/...' install-uos. Access the guest via virtctl console unifi / SSH once cloud-init adds a key.
  6. Verify UI: browse https://unifi.home.keithjackson.dev:11443, complete first-run wizard.
  7. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.1.54.
  8. Adopt a device: set its inform to http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
  9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
  10. (Optional) Enable the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.
Reference in new issue View Git Blame Copy Permalink