# UniFi Network controller (UniFi OS Server) — VM onboarding

> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
> or install cannot stall the app tier.
>
> **Status (verified 2026-08-26):**
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
>   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
>   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.

## What this is

A KubeVirt `VirtualMachine` running **UniFi OS Server (UOS)** — Ubiquiti's current self-host
product. The legacy `unifi` `.deb`/`unifi-network-server` is deprecated. UOS is a single installer
that runs on Ubuntu Server and bundles **UniFi OS + the Network application + its own embedded
datastore** inside a Podman container stack — so **no separate MongoDB companion is needed**
(this supersedes the earlier "unifi needs a Mongo container" note).

## Files

| File | Purpose |
|---|---|
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |

The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.

### Cloud-init delivery — two constraints learned the hard way

1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
   Secret deliberately.
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.

### Node prerequisites this VM forced onto the cluster

- **`nonroot-devices: true`** in the k3s server config (`infra/ansible/roles/k3s_server`) — the
  non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without
  it: `blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
  3-replica class made the import fail with insufficient storage.

## UOS install method (researched)

- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
  **`https://<host>:11443`**.
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.

### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

| Port | Proto | Use |
|---|---|---|
| 11443 | TCP | UOS web UI (modern default) |
| 8443 | TCP | Legacy Network UI (still served) |
| 8080 | TCP | **Device inform / adoption** |
| 8880 / 8843 | TCP | Guest portal HTTP / HTTPS |
| 6789 | TCP | Mobile throughput/speedtest |
| 3478 | UDP | **STUN** (device ⇄ controller) |
| 10001 | UDP | **L2 device discovery** |

## L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:

- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.

**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.

## Open questions / risks

1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
   import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
   Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
   fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
   migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
   installer and for optional remote access.
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
   masquerade boundary — see the L4 section. Expect explicit inform-host adoption.

## Remaining bring-up

Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.

1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
   Kustomization (not an app wave).
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
   `192.168.1.54`.
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
   complete the first-run wizard.
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
   connects (green). Test guest portal / throughput if used.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.

**Quick health check today:**
```bash
kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
```