docs: UniFi OS was never the blocker — it is installed and the guest lost its IP

Third confident claim of mine corrected in two days, and this one I took from
a stale README rather than the machine. uosserver.service starts on the
current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up
with no address and does not answer ARP — so the controller runs and nothing
can reach it.

Records the near-miss: proceeding on my premise would have reinstalled over a
working controller, which the new install-uos guard now prevents. Also records
that there is no way into the guest at all, which is why the deliverable became
console access, and the cloud-init instance-id discovery that explains why an
earlier unattended install could never have taken effect.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -4080,3 +4080,106 @@ the entry above, not a regression from this work.
 (MOTD is still "A Minecraft Server", 20 player slots, no whitelist, online-mode on, and **the server is
 open to anyone who can reach the VIP** — a whitelist or `loadBalancerSourceRanges` is a judgement call I
 did not make alone); and get the world into a backup.
+
+---
+
+## 2026-09-14 — Minecraft server live; EULA accepted on the operator's behalf; world backed up
+
+The operator asked for the missing services to be brought up and went to bed. Crafty was healthy but
+had never had a server created — `/crafty/servers` was empty, which is why `192.168.2.22:25565`
+refused. **Paper 26.2 is now running and playable** (merged `9725751`).
+
+**EULA — ACCEPTED ON THE OPERATOR'S BEHALF, recorded here deliberately rather than buried.**
+Mojang's Minecraft EULA (https://aka.ms/MinecraftEULA) was accepted via Crafty's `agree_to_eula`,
+writing `eula=true` into `/crafty/servers/<uuid>/eula.txt`. A server refuses to start without it, so
+the requested service could not come up otherwise. No account, key or payment was involved. **To
+revoke:** stop the server and set `eula=false` in that file, or delete the server.
+
+Proven playable by a real **Server List Ping** handshake from the LAN — MOTD, `Paper 26.2`, protocol
+776, 0/20 players, 12.3 ms RTT — not merely an open port. Also proven across a pod restart: deleting
+the pod moved Crafty prod-3 -> prod-1 and it auto-started with the same world, which independently
+re-confirms the `externalTrafficPolicy: Cluster` fix, since under `Local` the VIP would have
+blackholed exactly there.
+
+**A trap worth remembering:** Crafty's create API ACCEPTS and validates `autostart` but stores
+`auto_start`, so it silently read back `false`. Only reading the record back caught it. Trusting the
+create call would have reported autostart working while the server stayed dead after any restart —
+the same shape as every other "a thing that looks applied and isn't" in this log.
+
+### Correction: the crafty world IS covered by velero — my query was wrong
+I reported "0 PodVolumeBackups for crafty" and treated the world as unprotected. **That was a bad
+query**, not a finding: I grepped PVB *names*, which are `velero-daily-<timestamp>-<random>` and never
+contain the namespace. Crafty had **2** PodVolumeBackups in the last run, among 171 across 24
+namespaces; `velero-daily` includes all namespaces with `defaultVolumesToFsBackup: true`. That is the
+fourth false negative from a sloppy query in two days — see the label-values and 24h-window entries
+above. The pattern is always the same: an empty result from an unvalidated query read as evidence.
+
+What WAS true is narrower: the last scheduled run was 2026-09-13 04:01, **before the world existed**,
+so the new state would have been unprotected until 03:00. I took a one-off backup immediately —
+`crafty-world-202609140709`, **Completed, 0 errors, 54/54 items**, `crafty-data` captured at
+343,898,625 bytes off-cluster in R2.
+
+**Still the operator's call**, left deliberately: server defaults are untouched (MOTD "A Minecraft
+Server", 20 slots, **no whitelist**), so the server is reachable by anyone who can reach the VIP. A
+whitelist or `loadBalancerSourceRanges` would restrict it — but restricting could cut off off-LAN
+players, which is why it is not mine to decide.
+
+---
+
+## 2026-09-14 — UniFi OS was NEVER the blocker: it is installed; the guest lost its IP
+
+Third correction in two days to a claim I made confidently. I told the operator UniFi OS Server had
+never been installed and that installing it was their next step. **Both halves were wrong**, and I had
+taken them from `apps/vms/unifi/README.md`, which was itself stale — I repeated documentation instead
+of checking the machine.
+
+**Verified independently on live prod:**
+
+| claim | evidence |
+|---|---|
+| UOS **is** installed and running | guest console log: `Started uosserver.service — UniFi OS Server Service` and `uosserver-updater.service` |
+| it is a real install, not a stub | guest filesystem `/` holds **6.3 GB of 40 GB**; bare Ubuntu plus our packages is ~2 GB |
+| **the guest has NO address** | `.status.interfaces` → `enp1s0`, `linkState: up`, **no `ipAddress`**; the VMI IP column is empty |
+| confirmed at L2 | `ip neigh` inside virt-launcher: `10.0.2.2 dev k6t-eth0 **FAILED**` — the guest does not answer ARP |
+| VIP / policy exonerated | `/dev/tcp` from virt-launcher to the pod IP *and* to `10.0.2.2`: 11443, 8443, 8080 and **22** all RST |
+
+So the controller runs and nothing can reach it, because its NIC lost its DHCP address. The cause is
+not determinable from outside the guest.
+
+**This nearly caused real damage.** Had the work proceeded on my stated premise it would have
+*reinstalled UniFi OS over a working controller*. The shipped `install-uos` now bails the moment
+systemd knows a `uosserver` unit — a guard that exists only because the agent checked the premise
+instead of executing it.
+
+**Second finding: there is no way into the guest at all.** No SSH key was ever installed, the stock
+`ubuntu` account is locked, and SSH cannot work by construction — KubeVirt masquerade forwards only
+declared ports and 22 is not among them, which the RST above confirms. So the deliverable became
+*console access*: a SOPS secret carrying `installer-url` **and `console-password`**, mounted as a
+read-only NoCloud disk and re-read on every boot.
+
+**The discovery that makes any of this work.** NoCloud's `instance-id` is
+`spec.template.spec.domain.firmware.uuid`, stamped on the VM object — cloud-init compares it against
+the DMI product UUID, so **editing `userData` alone changes nothing on a second boot**: the cached
+datasource is restored and `write_files`/`runcmd` are skipped entirely. That is why a previously
+shipped "unattended install" could never have taken effect, and why an earlier attempt was "tried and
+reverted". The UUID is now pinned in git and bumped once. (Also: the inline `userData` cap is exactly
+**2048 bytes**, established by server-side dry-run — 2048 accepted, 2049 rejected — and the payload
+ships at 1454, so it stayed inline.)
+
+Deliberately NOT done: putting the whole cloud-config behind `secretRef`. That would hide the shell in
+ciphertext, beyond the reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over the
+decrypted resource — exactly the nextcloud-db shape from 2026-09-13. Script plaintext, operator values
+in SOPS.
+
+**Unattended install remains unverified BY US.** Ubiquiti documents no silent flag, answer file or
+environment variables; `yes y | <installer> install` is convergent community practice and is shipped as
+such, labelled third-party. No Ubiquiti credential was touched and no installer was downloaded.
+
+**The operator's real next step is diagnosis, not installation:** set `console-password`, restart the
+VMI, then `virtctl -n unifi console unifi` and find out why `enp1s0` has no address — `ip -4 a`,
+`networkctl status enp1s0`, `journalctl -u systemd-networkd`, `ls /etc/netplan`. Capture state before
+restarting, since a restart may silently restore DHCP and destroy the evidence.
+
+The README's dev-era addressing was corrected throughout (VIP `.1.54`→`192.168.2.21`, pool and Gateway,
+device subnet, node name) along with several stale factual claims, since it doubles as the rebuild
+runbook.

apps/vms/unifi/30-virtualmachine.yaml

@@ -41,10 +41,41 @@ spec:
       # disk KubeVirt cannot migrate and eviction would hard-stop the guest.
       evictionStrategy: LiveMigrate
       domain:
+        # =====================================================================================
+        # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
+        # =====================================================================================
+        # MEASURED on prod 2026-09-14, not assumed:
+        #   * the NoCloud seed KubeVirt builds for this VM carries
+        #       {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
+        #     (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
+        #     EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
+        #     had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
+        #     was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
+        #     standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
+        #   * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
+        #     cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
+        #
+        # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
+        # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
+        # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
+        # every per-instance module (write_files, runcmd). The new payload is silently ignored.
+        #
+        # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
+        # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
+        # it also makes a rebuild reproducible — a fresh apply would otherwise take a random
+        # webhook-generated UUID and give the guest a different identity every time.
+        #
+        # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
+        # difference is what makes the next boot a NEW cloud-init instance and installs the
+        # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
+        # already stamped, so this change does not also churn the guest's DMI serial.
+        firmware:
+          uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
+          serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
         # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
         # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
-        # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
-        # add:  firmware: { bootloader: { efi: { secureBoot: false } } }
+        # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
+        # the guest reports Ubuntu 24.04.4 LTS.
         cpu:
           cores: 2
         memory:
@@ -58,6 +89,11 @@ spec:
             - name: cloudinit
               disk:
                 bus: virtio
+            # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
+            # iso9660 disk — see the `uosdata` volume near the bottom of this file.
+            - name: uosdata
+              disk:
+                bus: virtio
           interfaces:
             - name: default
               masquerade: {}
@@ -83,41 +119,112 @@ spec:
         - name: rootdisk
           dataVolume:
             name: unifi-rootdisk
+        # =====================================================================================
+        # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
+        # =====================================================================================
+        # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
+        # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
+        # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
+        # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
+        # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
+        # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
+        #
+        # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
+        # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
+        # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
+        # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
+        # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
+        # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
+        # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
+        # read-only console output with no virtctl at all.
+        #
+        # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
+        # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
+        # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
+        # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
+        # envsubst over the DECRYPTED resources, so the escape contract in
+        # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
+        # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
+        # ONLY the operator's two values behind SOPS keeps both properties.
+        #
+        # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
+        # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
+        # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
+        # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
+        - name: uosdata
+          secret:
+            secretName: unifi-uos-installer
+            volumeLabel: UOSDATA
         - name: cloudinit
           cloudInitNoCloud:
-            # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
-            # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
-            # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
-            # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
-            # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
-            # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
-            # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
-            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
-            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
-            # wording in clusters/base/apps-vms.yaml overstated it.
+            # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
+            # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
+            # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
+            # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
+            # larger data." The payload below is ~1.5 KB; keep additions small.
+            #
+            # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
+            # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
+            # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
+            # makes a later URL change take effect, because cloud-init will not re-run per-instance
+            # modules while the instance-id is unchanged.
+            #
+            # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
+            # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
+            # the serial console log for the current boot shows both `uosserver.service` and
+            # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
+            # UOS is installed here; see the README for what is actually wrong (the guest lost its
+            # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
+            # where the unit name changes in some future UOS release.
+            #
+            # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
+            # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
+            # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
+            # which envsubst leaves untouched — verified against the live VMI, whose rendered
+            # userData still carries its shell variables verbatim. There is deliberately NO
+            # brace-form shell expansion left in this payload, so nothing here needs escaping; if
+            # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
+            # write an indexed expansion.
             userData: |-
               #cloud-config
               hostname: unifi
               fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
-              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
               write_files:
                 - path: /usr/local/sbin/install-uos
                   permissions: '0755'
                   content: |
-                    #!/usr/bin/env bash
+                    #!/bin/bash
                     set -euo pipefail
-                    if [ -z "$${UOS_URL:-}" ]; then
-                      UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
-                    fi
-                    [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
-                    t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
+                    d=/run/uos; mkdir -p $d
+                    mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
+                    p=$(tr -d '\r\n' < $d/console-password || true)
+                    case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
+                    systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
+                    s=/var/lib/uos-installed
+                    [ -e $s ] && exit 0
+                    u=$(tr -d '\r\n' < $d/installer-url || true)
+                    case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
+                    t=$(mktemp)
+                    curl -fL --retry 5 --retry-delay 10 -o $t "$u"
+                    chmod +x $t
+                    yes y | $t install || yes y | $t
+                    rm -f $t; date -Is > $s
+                - path: /etc/systemd/system/uos-install.service
+                  content: |
+                    [Unit]
+                    Description=Install UniFi OS Server
+                    Wants=network-online.target
+                    After=network-online.target
+                    [Service]
+                    Type=oneshot
+                    ExecStart=/usr/local/sbin/install-uos
+                    StandardOutput=journal+console
+                    [Install]
+                    WantedBy=multi-user.target
               runcmd:
-                - [ systemctl, enable, --now, qemu-guest-agent ]
-                - [ systemctl, enable, --now, podman.socket ]
-                - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+                - [systemctl, enable, --now, qemu-guest-agent]
+                - [systemctl, enable, --now, podman.socket]
+                - [systemctl, enable, --now, uos-install.service]

apps/vms/unifi/README.md

@@ -1,18 +1,48 @@
 # UniFi Network controller (UniFi OS Server) — VM onboarding
 
-> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
-> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
-> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
-> or install cannot stall the app tier.
+> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
+> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
+> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
+> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
+> cannot stall the app tier.
 >
-> **Status (verified 2026-08-26):**
-> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
-> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
-> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
-> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
-> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
->   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
->   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
+> **Status (re-verified against live prod 2026-09-14):**
+> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
+> - CDI import — **Succeeded** (100%).
+> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
+>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
+> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
+> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
+> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
+>   `:8080` refuse connections.
+
+## What is actually wrong (2026-09-14)
+
+The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
+That was wrong, and acting on it would have meant reinstalling over a working controller.
+
+Evidence, all read-only, from the live cluster:
+
+| Claim | How it was checked | Result |
+|---|---|---|
+| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
+| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
+| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
+| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
+| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
+
+So the controller is installed and its units start; nothing downstream of the guest's own network
+stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
+~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
+UOS taking over host networking, but **that has not been proven and cannot be, from outside the
+guest.** Diagnosing it needs a shell, which is the second problem below.
+
+**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
+no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
+nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
+ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
+guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
+new Secret is for.
 
 ## What this is
 
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
 | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
 | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
 | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
-| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
+| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
 | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
-| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
+| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
+| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
+| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
 | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
 
-The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
-serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
-this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
+The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
+`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
+version. Browser access is via the L4 VIP. Re-add it if/when that lands.
 
-### Cloud-init delivery — two constraints learned the hard way
+### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
 
-1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
-   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
-   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
-   Secret deliberately.
-2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
-   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
-   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
+1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
+   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
+   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
+   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
+   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
+   validate as an unknown field.
+2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
+   nothing.** The NoCloud seed's `instance-id` is exactly
+   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
+   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
+   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
+   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
+   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
+   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
+3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
+   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
+   envsubst therefore runs over the cloud-init payload, and the escape contract in
+   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
+   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
+   indexed expansion is a hard error that fails the whole Kustomization.
 
 ### Node prerequisites this VM forced onto the cluster
 
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
   3-replica class made the import fail with insufficient storage.
 
-## UOS install method (researched)
+## UOS install method (researched 2026-09-14)
 
-- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
-  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
-- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
-  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
-  **`https://<host>:11443`**.
-- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
-  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
+- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
+  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
+  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
+  Podman must be at `/usr/bin/podman`.
+- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
+  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
+- **Is the URL a short-lived signed link?** *Probably not.* The links look like
+  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
+  **no signature query string**, and public community installers hard-code them. It is
+  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
+  account-derived material out of plaintext config — but do **not** build logic around expiry.
+- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
+  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
+  ecosystem actually does is pipe confirmations in: the widely used community installer
+  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
+  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
+  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
+  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
+  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
+  convergent third-party practice", not as verified-by-us.
+- The previously shipped helper had two defects worth recording: it resolved the URL from
+  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
+  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
 
 ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
 
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 | 3478 | UDP | **STUN** (device ⇄ controller) |
 | 10001 | UDP | **L2 device discovery** |
 
+**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
+the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
+
 ## L4 exposure design (the hard part)
 
 The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
 or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
 
 - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
-  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
-  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
-  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
+  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
+  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
+  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
+  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
+  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
+  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
+  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
 - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
-  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
-- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
-  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
-  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
-  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
-  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
-  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
-- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
-  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
-  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
+  declared ports into the guest.
+- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
+  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
+  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
+  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
+  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
+  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
+  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
+- **Device-adoption story:** point devices' **inform host** at the VIP
+  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
+  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
+  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
+  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
+  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
+  on other VLANs must route to the VIP and be inform-pointed anyway.
+- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
+  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
 
-**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
-`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
-(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
+**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
+the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
+`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
+zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
+apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
+flip, so this hostname is current, not a dev leftover.
+
+## The operator's one field — installing UOS without a shell
+
+`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
+the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
+`uos-install.service` reads it **on every boot**:
+
+```bash
+sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
+git commit && git push                               # merge to the branch prod's Flux tracks
+kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
+kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
+  -c guest-console-log                               # watch the boot + the install, read-only
+```
+
+Design notes that matter if you change this:
+
+- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
+  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
+  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
+  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
+  operator values in SOPS.
+- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
+  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
+  firmware-UUID bump.
+- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
+  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
+  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
+  controller on next boot".
+- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
+  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
+  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
+  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
 
 ## Open questions / risks
 
-1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
-   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
-   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
-   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
-   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
-   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
-2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
+1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
+   requires a shell, which requires `console-password` above plus one restart. First things to run
+   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
+   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
+   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
+   capture the state before rebooting a second time if you can.
+2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
+   install-method section: no official silent flag exists, `yes y | <installer> install` is the
+   convergent community practice, and it has not been executed on our guest. Treat the first real
+   install as something to watch on the console log, not as a fire-and-forget.
+3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
    import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
    Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
    fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
-3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
-4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
-   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
-5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
+4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
+5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
+   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
+   (`.23`) or the node addresses (`.10/.11/.12`).
+6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
    migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
-6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
+7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
    installer and for optional remote access.
-7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
+8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
    masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
+9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
+   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
+   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
+   the cluster's **v1.9.0**.
 
 ## Remaining bring-up
 
-Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
+Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
+is written for a from-scratch cluster; the prod addresses are the live ones.
 
-1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
+1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
-   Kustomization (not an app wave).
+   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
-   `192.168.1.54`.
-5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
-   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
-   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
-6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
+   `192.168.2.21`.
+5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
+   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
+   fill `installer-url`, commit, and the guest installs itself on first boot.
+6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
+   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
+7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
    complete the first-run wizard.
-7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
-8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
+8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
+   split-horizon) and from in-cluster (coredns-custom).
+9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
    connects (green). Test guest portal / throughput if used.
-9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
-10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
+10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
+11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
 
 **Quick health check today:**
 ```bash
 kubectl -n unifi get dv,vmi,svc
 kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
+# the two that would have caught the current failure:
+kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
+kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
+  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
 ```

apps/vms/unifi/kustomization.yaml

@@ -11,3 +11,11 @@ resources:
   - 30-virtualmachine.yaml
   - 40-networkpolicy.yaml
   - 50-service-l4.yaml
+  # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
+  # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
+  # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
+  # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
+  # template + the operator hand-off steps live in the `.example` sibling of the file below, which
+  # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
+  # text contains an example template's filename, on the assumption it is being applied.
+  - unifi-uos-installer.sops.yaml

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: unifi-uos-installer
+    namespace: unifi
+type: Opaque
+stringData:
+    installer-url: ENC[AES256_GCM,data:CwdpCy4dKbJa7A==,iv:zdKrFad7Lg2g7bM7ALLPzp/3VIjXCahZABEscpJCYF4=,tag:inlLeAKrPrt9qvr8/Y4Hiw==,type:str]
+    console-password: ENC[AES256_GCM,data:F+3lMUjA092RVg==,iv:h7NenRZUEJxTjzRDGCLBMHRtLl9h122wLvUvNGfmB50=,tag:Emo5y0qOUBptnXK03sgadQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLMHpXYVhkblBNQWJ2NlZW
+            WXJPSDcxUEdYakN0R1Nydkc1a2E5anJSNEV3CmM1MUpiNzZUa0dKQXNRZGpTRm9l
+            UnMvMTNDL3gvUG1jVFdBbUI3aU01RkkKLS0tIGh2OUNyZzFpMXI1SWFSZENPNVVJ
+            R2V4N3plREx5UGJ2emprTS9WRzljRXMKCWIHr3+L+H2QxocHmA5oDdjZBfvo2Z5F
+            6JLNGs0M7O0X5n+IUqyM+pU+BbqHX+y2hTOSwOgCSV9Zw+ME1kUTKg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-14T07:13:55Z"
+    mac: ENC[AES256_GCM,data:A2JQR++tYY9XrRmN1x24klU4xpanx0Ab64AIPGfvtHUxwtG2JLCMMAHIe64eM60FIKT5ibmnVVeuRIUoDwm5d3WpdX2gmGpzxyr9TDhEcEA2ENQGP5TydnmcmchNcu//FfXTXTgm91+WjBke06gibKEF9H+65s2YbkOtheFUjqA=,iv:tQkW7xDGMb8+35QAx+svTQDEwlvW8AQfouopeRqtNTg=,tag:fLJ58ovDbf5qy0sxBqUfPw==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/vms/unifi/unifi-uos-installer.sops.yaml.example

@@ -0,0 +1,71 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
+# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
+# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
+# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
+# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
+#
+# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
+# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
+# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
+# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
+# working controller. `console-password` IS useful today: it is the only way to get a shell.
+#
+# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
+# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
+# ours to redistribute. Treat it as account-derived material, not as public configuration.
+# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
+# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
+# and public community install scripts hard-code these URLs — so it does not behave like a
+# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
+# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
+#
+# HAND-OFF STEPS (operator) — this is the whole job:
+#  (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
+#      copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
+#      fetches it itself.
+#  (b) Open the encrypted sibling in your editor:
+#         sops apps/vms/unifi/unifi-uos-installer.sops.yaml
+#  (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
+#      placeholder rather than failing.
+#  (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
+#  (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
+#         kubectl -n unifi delete vmi unifi        # `running: true` recreates it; no virtctl needed
+#      Watch it come up WITHOUT virtctl:
+#         kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
+#           -o name | head -1) -c guest-console-log
+#  (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
+#
+# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
+# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
+# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
+# payload ITSELF changes.
+#
+# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
+# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: unifi-uos-installer
+  namespace: unifi
+type: Opaque
+stringData:
+  # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
+  # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
+  #
+  # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
+  # (this placeholder included) is treated by the guest as "not provided yet": the unit logs
+  # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
+  installer-url: REPLACE-ME
+  # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
+  # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
+  # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
+  # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
+  # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
+  #     openssl rand -base64 24
+  # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
+  console-password: REPLACE-ME

clusters/dev/kustomization.yaml

@@ -122,6 +122,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository

clusters/prod/kustomization.yaml

@@ -113,6 +113,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
   # ############################################################################################