Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(unifi): SOPS-delivered UOS installer URL + per-boot install unit; correct the README

Browse Source 
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.

Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
  * inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
    rejected 2049; the payload here is 1454.
  * `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
    disk + third disk were accepted by a server-side dry-run of the real manifest.
  * the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
    noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
    userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
    write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.

secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.

CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.

There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.

README corrections (it doubles as the rebuild runbook and was misdirecting):
  * VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
    Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
    All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
  * "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
  * "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
    contract is in force.
  * "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
    over (an unverified firmware API, and no `install` subcommand).
  * "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
  * externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
  * 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
  * userDataSecretRef -> the YAML key is `secretRef`.
  * 90-httproute-optional.yaml referenced as if it existed; it was dropped.

Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.

The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-14 02:19:13 -05:00
1 parent 83f13f4b0a
commit 12d4e0ae40
7 files changed
+477 -100
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/vms/unifi/README.md
+202 -74
View File
@@ -1,18 +1,48 @@
# UniFi Network controller (UniFi OS Server) — VM onboarding
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
> or install cannot stall the app tier.
> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
> cannot stall the app tier.
>
> **Status (verified 2026-08-26):**
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
> **Status (re-verified against live prod 2026-09-14):**
> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
> - CDI import — **Succeeded** (100%).
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
> `:8080` refuse connections.
## What is actually wrong (2026-09-14)
The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
That was wrong, and acting on it would have meant reinstalling over a working controller.
Evidence, all read-only, from the live cluster:
| Claim | How it was checked | Result |
|---|---|---|
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
So the controller is installed and its units start; nothing downstream of the guest's own network
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
guest.** Diagnosing it needs a shell, which is the second problem below.
**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
new Secret is for.
## What this is
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
version. Browser access is via the L4 VIP. Re-add it if/when that lands.
### Cloud-init delivery — two constraints learned the hard way
### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
Secret deliberately.
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
validate as an unknown field.
2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
nothing.** The NoCloud seed's `instance-id` is exactly
`spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
(so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
`runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
to force a re-run. See the long comment in `30-virtualmachine.yaml`.
3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
envsubst therefore runs over the cloud-init payload, and the escape contract in
`clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
indexed expansion is a hard error that fails the whole Kustomization.
### Node prerequisites this VM forced onto the cluster
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
3-replica class made the import fail with insufficient storage.
## UOS install method (researched)
## UOS install method (researched 2026-09-14)
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
`ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
**`https://<host>:11443`**.
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
(the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
Podman must be at `/usr/bin/podman`.
- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
- **Is the URL a short-lived signed link?** *Probably not.* The links look like
`https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
**no signature query string**, and public community installers hard-code them. It is
account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
account-derived material out of plaintext config — but do **not** build logic around expiry.
- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
ecosystem actually does is pipe confirmations in: the widely used community installer
(`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
`yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
(no URL has been supplied, and UOS is already installed here), so treat it as "established by
convergent third-party practice", not as verified-by-us.
- The previously shipped helper had two defects worth recording: it resolved the URL from
`fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
`unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
| 3478 | UDP | **STUN** (device ⇄ controller) |
| 10001 | UDP | **L2 device discovery** |
**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
## L4 exposure design (the hard part)
The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
(`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
`default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
/ `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
— via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
`minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
declared ports into the guest.
- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
- **Device-adoption story:** point devices' **inform host** at the VIP
(`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
`set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover.
## The operator's one field — installing UOS without a shell
`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
`uos-install.service` reads it **on every boot**:
```bash
sops apps/vms/unifi/unifi-uos-installer.sops.yaml # fill installer-url and/or console-password
git commit && git push # merge to the branch prod's Flux tracks
kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
-c guest-console-log # watch the boot + the install, read-only
```
Design notes that matter if you change this:
- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
operator values in SOPS.
- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
firmware-UUID bump.
- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
controller on next boot".
- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
`apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
`clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
## Open questions / risks
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
requires a shell, which requires `console-password` above plus one restart. First things to run
on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
`ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
capture the state before rebooting a second time if you can.
2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
install-method section: no official silent flag exists, `yes y | <installer> install` is the
convergent community practice, and it has not been executed on our guest. Treat the first real
install as something to watch on the console log, not as a fire-and-forget.
3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
(`.23`) or the node addresses (`.10/.11/.12`).
6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
installer and for optional remote access.
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
`kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
`virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
the cluster's **v1.9.0**.
## Remaining bring-up
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
is written for a from-scratch cluster; the prod addresses are the live ones.
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
Kustomization (not an app wave).
Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
`192.168.1.54`.
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
`192.168.2.21`.
5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
`uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
fill `installer-url`, commit, and the guest installs itself on first boot.
6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
complete the first-run wizard.
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
split-horizon) and from in-cluster (coredns-custom).
9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
connects (green). Test guest portal / throughput if used.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
**Quick health check today:**
```bash
kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
# the two that would have caught the current failure:
kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}' # expect an ipAddress
kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
-c compute -- ip neigh # expect 10.0.2.2 REACHABLE
```