home-network-k3s-impl/apps/vms/unifi/README.md

UniFi Network controller (UniFi OS Server) — VM onboarding

LIVE — but the controller is unreachable, and NOT for the reason this file used to claim. Reconciled by the non-gating apps-vms Flux Kustomization (clusters/base/apps-vms.yaml, dependsOn: platform-config + platform-config-cdi, wait: false). VMs are a separate lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install cannot stall the app tier.

Status (re-verified against live prod 2026-09-14):

• longhorn-vm StorageClass — live; unifi-rootdisk PVC Bound, 40Gi, RWX, Block.
• CDI import — Succeeded (100%).
• VM — Running on k3s-prod-1, guest agent connected, Ubuntu 24.04.4 LTS, LiveMigratable=True / StorageLiveMigratable=True. VMI age 34h, VM 8d.
• L4 LoadBalancer VIP 192.168.2.21 — allocated and held on both Services.
• UniFi OS Server IS INSTALLED. This reverses the previous entry here. See below.
• The guest has no IP address. That — not a missing install — is why :11443, :8443 and :8080 refuse connections.

What is actually wrong (2026-09-14)

The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu". That was wrong, and acting on it would have meant reinstalling over a working controller.

Evidence, all read-only, from the live cluster:

Claim	How it was checked	Result
UOS is installed	kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log	the current boot starts uosserver.service — UniFi OS Server Service and uosserver-updater.service
…and it is a real install, not a stub	KubeVirt guest-agent filesystemlist subresource	/ is 6.3 GB used of 40 GB — a bare Ubuntu cloud image with our packages is ~2 GB
the guest has no IP	kubectl -n unifi get vmi unifi -o json → .status.interfaces	enp1s0, linkState: up, no ipAddress field at all; the kubectl get vmi IP column is empty
…confirmed at the link layer	kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh	10.0.2.2 dev k6t-eth0 FAILED — the guest does not answer ARP for its masquerade address
the ports are dead end-to-end	/dev/tcp probes from inside virt-launcher, to the pod IP and to 10.0.2.2 directly	11443 / 8443 / 8080 / 22 all refused — i.e. the VIP and the CNP are exonerated

So the controller is installed and its units start; nothing downstream of the guest's own network stack can work. The interface was fine at boot (the console log reaches network-online.target at ~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is UOS taking over host networking, but that has not been proven and cannot be, from outside the guest. Diagnosing it needs a shell, which is the second problem below.

There is currently no way into this guest. The shipped cloud-init installs no SSH key and sets no password, so the stock ubuntu account is locked and a serial console reaches a login prompt nobody can answer. SSH would not rescue it either: the masquerade interface forwards only the ports declared in 30-virtualmachine.yaml, and 22 is not one of them — the probe above shows the guest RSTs on 22 exactly as it does on 11443. Closing that hole is what console-password in the new Secret is for.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File	Purpose
namespace.yaml	unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml	R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml	CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (40Gi). First real CDI http import on the cluster
30-virtualmachine.yaml	The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, a pinned firmware UUID (= the cloud-init instance-id), the UOSDATA secret disk, and a minimal inline cloud-init that installs install-uos + a per-boot systemd unit
40-networkpolicy.yaml	Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml	Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP ${UNIFI_L4_VIP} (prod 192.168.2.21); external-dns A record
unifi-uos-installer.sops.yaml	SOPS-encrypted. The operator's UOS installer URL + guest console password, mounted into the guest as a disk
unifi-uos-installer.sops.yaml.example	Plaintext template + the operator hand-off steps (never applied, never encrypted)
kustomization.yaml	The reconciled resource list (referenced by apps/vms/kustomization.yaml)

The optional L7 HTTPRoute was dropped, and the file does not exist: UOS serves HTTPS on :11443, so a Gateway route needs a BackendTLSPolicy that is not confirmed on this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.

Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)

1. Inline userData is capped at 2048 bytes. MEASURED, not quoted: a server-side dry-run apply of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data." The shipped payload is ~1.45 KB. The API field for the escape hatch is spelled secretRef (the Go name is UserDataSecretRef); userDataSecretRef is not a YAML key and trying it will silently validate as an unknown field.
2. The firmware UUID is the cloud-init instance-id, and editing userData alone changes nothing. The NoCloud seed's instance-id is exactly spec.template.spec.domain.firmware.uuid, which KubeVirt's webhook stamps onto the VM object (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI product UUID. Same id → cached datasource restored, new payload never read, write_files and runcmd skipped. This is why the previously shipped "unattended install" could never have taken effect on a second boot. The UUID is now pinned in git, with bumping it as the documented way to force a re-run. See the long comment in 30-virtualmachine.yaml.
3. apps-vms DOES use postBuild.substituteFrom. This file used to say it must not — that has been false since 2026-08-31, when ${UNIFI_L4_VIP} landed in 50-service-l4.yaml. Flux's envsubst therefore runs over the cloud-init payload, and the escape contract in clusters/base/apps-vms.yaml is in force. Bare $name and $(cmd) are left alone (verified against the live VMI's rendered userData); brace-form shell expansion must be escaped; an indexed expansion is a hard error that fails the whole Kustomization.

Node prerequisites this VM forced onto the cluster

• nonroot-devices: true in the k3s server config (infra/ansible/roles/k3s_server) — the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without it: blockdev: cannot open /dev/cdi-block-volume: Permission denied.
• longhorn-scratch — a single-replica StorageClass for CDI scratch space; the default 3-replica class made the import fail with insufficient storage.

UOS install method (researched 2026-09-14)

• Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give 2 vCPU / 4 GiB / 40Gi (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong). Podman must be at /usr/bin/podman.
• Install: obtain the download link from ui.com/download/software/unifi-os-server while signed in to a (free) Ubiquiti account, chmod +x, run as root. UI then on https://<host>:11443.
• Is the URL a short-lived signed link? Probably not. The links look like https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64 with no signature query string, and public community installers hard-code them. It is account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps account-derived material out of plaintext config — but do not build logic around expiry.
• Can it run unattended? Yes in practice, no officially — and this is the honest answer. Ubiquiti documents no --silent flag, no answer file and no environment variables. What the ecosystem actually does is pipe confirmations in: the widely used community installer (dgodibadze/UniFi_OS_Server) runs yes y | "$file" install, and vendor-adjacent walkthroughs run either ./<binary> install or a bare sudo ./<binary>. install-uos therefore does yes y | $t install || yes y | $t — the documented-by-practice form first, the bare form as a fallback for builds that take no subcommand. This has not been executed end-to-end on our guest (no URL has been supplied, and UOS is already installed here), so treat it as "established by convergent third-party practice", not as verified-by-us.
• The previously shipped helper had two defects worth recording: it resolved the URL from fw-update.ui.com/api/firmware-latest, an endpoint nothing in this repo ever verified lists unifi-os-server, and it invoked the binary with no install subcommand.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port	Proto	Use
11443	TCP	UOS web UI (modern default)
8443	TCP	Legacy Network UI (still served)
8080	TCP	Device inform / adoption
8880 / 8843	TCP	Guest portal HTTP / HTTPS
6789	TCP	Mobile throughput/speedtest
3478	UDP	STUN (device ⇄ controller)
10001	UDP	L2 device discovery

22/TCP is deliberately absent, and that is load-bearing: a masquerade interface forwards only the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

• One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned through ${UNIFI_L4_VIP} (lbipam.cilium.io/ips) — 192.168.2.21 on prod. The VIP comes from the existing default-pool, whose live prod block is 192.168.2.20–192.168.2.30 with 192.168.2.20 held by the shared Gateway (cilium-gateway-shared), and is ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config. (The 192.168.1.53–.60 / .53 figures this section used to quote were dev addressing and are gone; dev has been decommissioned. Neighbours in the prod pool today: .22 crafty, .23 pihole-dns.)
• selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest.
• externalTrafficPolicy: Cluster, corrected 2026-09-14 after an 8-day LAN outage. This file used to advertise Local "to preserve the LAN client source IP". Both halves were false here: Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the leases sat on k3s-prod-2 while the VM ran on k3s-prod-1, so Local had nowhere to forward), and KubeVirt's masquerade binding had already NATed the client IP away before the guest ever saw it. The full reasoning — including why 40-networkpolicy.yaml needs no remote-node entry and why no loadBalancerSourceRanges was added — is in 50-service-l4.yaml.
• Device-adoption story: point devices' inform host at the VIP (http://192.168.2.21:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.2.0/24, LAN_CIDR), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
• Reusable pattern: L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. crafty (.22) is the same shape and is live.

UI access: https://unifi.home.keithjackson.dev:11443. That name resolves to the VIP and not to the Gateway on purpose: both platform/coredns/coredns-custom.yaml (in-cluster) and apps/wave-0/pihole.yaml (LAN split-horizon) carve unifi.${BASE_DOMAIN} out of the zone-wide ${GATEWAY_VIP} answer and point it at ${UNIFI_L4_VIP}. ${BASE_DOMAIN} on prod is the apex home.keithjackson.dev (confirmed live in the cluster-vars ConfigMap) after the 2026-09-13 flip, so this hostname is current, not a dev leftover.

The operator's one field — installing UOS without a shell

unifi-uos-installer.sops.yaml is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into the guest as a read-only iso9660 disk labelled UOSDATA (one file per key), and an in-guest uos-install.service reads it on every boot:

sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
git commit && git push                               # merge to the branch prod's Flux tracks
kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
  -c guest-console-log                               # watch the boot + the install, read-only

Design notes that matter if you change this:

• The Secret is a disk, not cloudInitNoCloud.secretRef. secretRef would lift the 2048-byte cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over decrypted resources) — the exact shape of the 2026-09-13 nextcloud-db incident. Script in plaintext, operator values in SOPS.
• A URL change needs no UUID bump. The disk is rebuilt from the Secret on every VMI start and the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the firmware-UUID bump.
• install-uos will not reinstall over a working controller. It exits as soon as systemd knows a uosserver.service, with /var/lib/uos-installed as a weaker second guard. Given the live guest already has UOS, this guard is the difference between "safe to merge" and "destroys the controller on next boot".
• apps-vms had no SOPS decryption stanza — this is the first encrypted content under apps/vms/, so the stanza was added to both clusters/dev/kustomization.yaml and clusters/prod/kustomization.yaml. Without it kustomize hands Flux the raw SOPS document and the guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.

Open questions / risks

1. The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator. Diagnosis requires a shell, which requires console-password above plus one restart. First things to run on the console: ip -4 a, networkctl status enp1s0, journalctl -u systemd-networkd, ls /etc/netplan, and systemctl status uosserver. A plain restart may well restore DHCP on its own (the address survived the first ~34h of this boot's life), but that would hide the cause, so capture the state before rebooting a second time if you can.
2. Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed. See the install-method section: no official silent flag exists, yes y | <installer> install is the convergent community practice, and it has not been executed on our guest. Treat the first real install as something to watch on the console log, not as a fire-and-forget.
3. CDI http import (LOW) — RESOLVED in practice. The import succeeded (first http/qcow2 CDI import on this cluster). Still no checksum on an http source, and the URL tracks a moving Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level fixes (nonroot-devices, longhorn-scratch) — see above.
4. Boot firmware (LOW) — RESOLVED. Boots as shipped; running Ubuntu 24.04.4 LTS.
5. VIP 192.168.2.21 (LOW) — RESOLVED. Allocated and held by both Services, inside the live prod pool 192.168.2.20–.30, not colliding with the Gateway (.20), crafty (.22), pihole-dns (.23) or the node addresses (.10/.11/.12).
6. Longhorn RWX prerequisite (was BLOCKING) — RESOLVED. longhorn-vm is live (RWX / Block / migratable); the VMI reports LiveMigratable=True and StorageLiveMigratable=True.
7. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.
8. Broadcast adoption (LOW, by design). L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
9. No virtctl in the devcontainer (LOW). Restart and read-only console both work with plain kubectl (above). An interactive console — the thing console-password unlocks — needs virtctl on the operator's path; it is a single static binary from the KubeVirt release matching the cluster's v1.9.0.

Remaining bring-up

Steps 1–5 are done. The sequence is kept because it is also the recovery/rebuild runbook, so it is written for a from-scratch cluster; the prod addresses are the live ones.

1. longhorn-vm StorageClass live; a free address in the LB pool. Done (192.168.2.21).
2. Wire into a reconciled path — done, via apps/vms/ + the non-gating apps-vms Kustomization (not an app wave), including its SOPS decryption stanza in both overlays.
3. CDI import — done: kubectl -n unifi get dv unifi-rootdisk → Succeeded.
4. VM + VIP — done: kubectl -n unifi get vmi unifi → Running; both Services show 192.168.2.21.
5. Install UOS in the guest — done (it was already done when this file claimed otherwise; uosserver.service starts on every boot). On a REBUILD this is where the Secret does the work: fill installer-url, commit, and the guest installs itself on first boot.
6. → NEXT, and it is the operator's: fill console-password, restart the VM once, get on the console and find out why enp1s0 has no address. Until that is fixed nothing below can pass.
7. Verify the UI: https://unifi.home.keithjackson.dev:11443 (or https://192.168.2.21:11443), complete the first-run wizard.
8. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.2.21 from a LAN client (pihole split-horizon) and from in-cluster (coredns-custom).
9. Adopt a device: set its inform to http://192.168.2.21:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
11. (Optional) Re-add the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.

Quick health check today:

kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
# the two that would have caught the current failure:
kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE