home-network-k3s-impl/apps/vms/unifi/README.md

# UniFi Network controller (UniFi OS Server) — VM onboarding

> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
> cannot stall the app tier.
>
> **Status (re-verified against live prod 2026-09-14):**
> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
> - CDI import — **Succeeded** (100%).
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
>   `:8080` refuse connections.

## What is actually wrong (2026-09-14)

The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
That was wrong, and acting on it would have meant reinstalling over a working controller.

Evidence, all read-only, from the live cluster:

| Claim | How it was checked | Result |
|---|---|---|
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |

So the controller is installed and its units start; nothing downstream of the guest's own network
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
guest.** Diagnosing it needs a shell, which is the second problem below.

**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
new Secret is for.

## What this is

A KubeVirt `VirtualMachine` running **UniFi OS Server (UOS)** — Ubiquiti's current self-host
product. The legacy `unifi` `.deb`/`unifi-network-server` is deprecated. UOS is a single installer
that runs on Ubuntu Server and bundles **UniFi OS + the Network application + its own embedded
datastore** inside a Podman container stack — so **no separate MongoDB companion is needed**
(this supersedes the earlier "unifi needs a Mongo container" note).

## Files

| File | Purpose |
|---|---|
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |

The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
version. Browser access is via the L4 VIP. Re-add it if/when that lands.

### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)

1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
   validate as an unknown field.
2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
   nothing.** The NoCloud seed's `instance-id` is exactly
   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
   envsubst therefore runs over the cloud-init payload, and the escape contract in
   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
   indexed expansion is a hard error that fails the whole Kustomization.

### Node prerequisites this VM forced onto the cluster

- **`nonroot-devices: true`** in the k3s server config (`infra/ansible/roles/k3s_server`) — the
  non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without
  it: `blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
  3-replica class made the import fail with insufficient storage.

## UOS install method (researched 2026-09-14)

- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
  Podman must be at `/usr/bin/podman`.
- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
- **Is the URL a short-lived signed link?** *Probably not.* The links look like
  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
  **no signature query string**, and public community installers hard-code them. It is
  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
  account-derived material out of plaintext config — but do **not** build logic around expiry.
- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
  ecosystem actually does is pipe confirmations in: the widely used community installer
  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
  convergent third-party practice", not as verified-by-us.
- The previously shipped helper had two defects worth recording: it resolved the URL from
  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.

### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

| Port | Proto | Use |
|---|---|---|
| 11443 | TCP | UOS web UI (modern default) |
| 8443 | TCP | Legacy Network UI (still served) |
| 8080 | TCP | **Device inform / adoption** |
| 8880 / 8843 | TCP | Guest portal HTTP / HTTPS |
| 6789 | TCP | Mobile throughput/speedtest |
| 3478 | UDP | **STUN** (device ⇄ controller) |
| 10001 | UDP | **L2 device discovery** |

**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.

## L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:

- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
  declared ports into the guest.
- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
- **Device-adoption story:** point devices' **inform host** at the VIP
  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
  on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.

**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover.

## The operator's one field — installing UOS without a shell

`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
`uos-install.service` reads it **on every boot**:

```bash
sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
git commit && git push                               # merge to the branch prod's Flux tracks
kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
  -c guest-console-log                               # watch the boot + the install, read-only
```

Design notes that matter if you change this:

- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
  operator values in SOPS.
- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
  firmware-UUID bump.
- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
  controller on next boot".
- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.

## Open questions / risks

1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
   requires a shell, which requires `console-password` above plus one restart. First things to run
   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
   capture the state before rebooting a second time if you can.
2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
   install-method section: no official silent flag exists, `yes y | <installer> install` is the
   convergent community practice, and it has not been executed on our guest. Treat the first real
   install as something to watch on the console log, not as a fire-and-forget.
3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
   import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
   Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
   fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
   (`.23`) or the node addresses (`.10/.11/.12`).
6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
   migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
   installer and for optional remote access.
8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
   masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
   the cluster's **v1.9.0**.

## Remaining bring-up

Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
is written for a from-scratch cluster; the prod addresses are the live ones.

1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
   `192.168.2.21`.
5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
   fill `installer-url`, commit, and the guest installs itself on first boot.
6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
   complete the first-run wizard.
8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
   split-horizon) and from in-cluster (coredns-custom).
9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
   connects (green). Test guest portal / throughput if used.
10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.

**Quick health check today:**
```bash
kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
# the two that would have caught the current failure:
kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
```