From 12d4e0ae40d8219eb3d7203f04c73d0aac98791a Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 14 Sep 2026 02:16:42 -0500 Subject: [PATCH] feat(unifi): SOPS-delivered UOS installer URL + per-boot install unit; correct the README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml (SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url` and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit and a VM restart. Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory: * inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and rejected 2049; the payload here is 1454. * `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret disk + third disk were accepted by a server-side dry-run of the real manifest. * the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands. secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was added to both overlays. CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of 40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/ 8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service — without that guard this change would have reinstalled over a working controller. There was also no way into the guest at all: no key, no password, and 22 is not a declared masquerade port so SSH cannot work by construction. Hence console-password. README corrections (it doubles as the rebuild runbook and was misdirecting): * VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1. All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR). * "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table. * "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape contract is in force. * "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice over (an unverified firmware API, and no `install` subcommand). * "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded. * externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml). * 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB. * userDataSecretRef -> the YAML key is `secretRef`. * 90-httproute-optional.yaml referenced as if it existed; it was dropped. Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars. `yes y | install` is the convergent community practice and is what ships here, with a bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also does not look signed or time-limited — it is account-derived and per-release; it is in SOPS anyway, but nothing should depend on expiry. The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the 34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart. Co-Authored-By: Claude Opus 5 --- apps/vms/unifi/30-virtualmachine.yaml | 159 ++++++++-- apps/vms/unifi/README.md | 276 +++++++++++++----- apps/vms/unifi/kustomization.yaml | 8 + apps/vms/unifi/unifi-uos-installer.sops.yaml | 29 ++ .../unifi-uos-installer.sops.yaml.example | 71 +++++ clusters/dev/kustomization.yaml | 17 ++ clusters/prod/kustomization.yaml | 17 ++ 7 files changed, 477 insertions(+), 100 deletions(-) create mode 100644 apps/vms/unifi/unifi-uos-installer.sops.yaml create mode 100644 apps/vms/unifi/unifi-uos-installer.sops.yaml.example diff --git a/apps/vms/unifi/30-virtualmachine.yaml b/apps/vms/unifi/30-virtualmachine.yaml index 6d99442..fec037e 100644 --- a/apps/vms/unifi/30-virtualmachine.yaml +++ b/apps/vms/unifi/30-virtualmachine.yaml @@ -41,10 +41,41 @@ spec: # disk KubeVirt cannot migrate and eviction would hard-stop the guest. evictionStrategy: LiveMigrate domain: + # ===================================================================================== + # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT. + # ===================================================================================== + # MEASURED on prod 2026-09-14, not assumed: + # * the NoCloud seed KubeVirt builds for this VM carries + # {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"} + # (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was + # EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook + # had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid + # was b532fd7c-…, a different value). It is not a hash of the VM name either: the four + # standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else. + # * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the + # cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID. + # + # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have + # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an + # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips + # every per-instance module (write_files, runcmd). The new payload is silently ignored. + # + # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest + # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning + # it also makes a rebuild reproducible — a fresh apply would otherwise take a random + # webhook-generated UUID and give the guest a different identity every time. + # + # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that + # difference is what makes the next boot a NEW cloud-init instance and installs the + # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook + # already stamped, so this change does not also churn the guest's DMI serial. + firmware: + uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11" + serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18" # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable, # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which - # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot, - # add: firmware: { bootloader: { efi: { secureBoot: false } } } + # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped; + # the guest reports Ubuntu 24.04.4 LTS. cpu: cores: 2 memory: @@ -58,6 +89,11 @@ spec: - name: cloudinit disk: bus: virtio + # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only + # iso9660 disk — see the `uosdata` volume near the bottom of this file. + - name: uosdata + disk: + bus: virtio interfaces: - name: default masquerade: {} @@ -83,41 +119,112 @@ spec: - name: rootdisk dataVolume: name: unifi-rootdisk + # ===================================================================================== + # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK. + # ===================================================================================== + # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose; + # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and + # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with + # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https + # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the + # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file. + # + # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock + # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key, + # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared + # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the + # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the + # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi` + # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives + # read-only console output with no virtctl at all. + # + # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the + # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two + # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the + # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild + # envsubst over the DECRYPTED resources, so the escape contract in + # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact + # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting + # ONLY the operator's two values behind SOPS keeps both properties. + # + # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the + # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below + # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see + # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed. + - name: uosdata + secret: + secretName: unifi-uos-installer + volumeLabel: UOSDATA - name: cloudinit cloudInitNoCloud: - # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via - # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected - # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from - # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443). - # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for - # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData. - # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits - # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone - # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write - # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace") - # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against - # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal - # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not" - # wording in clusters/base/apps-vms.yaml overstated it. + # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the + # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an + # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with + # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for + # larger data." The payload below is ~1.5 KB; keep additions small. + # + # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is + # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos` + # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what + # makes a later URL change take effect, because cloud-init will not re-run per-instance + # modules while the instance-id is unchanged. + # + # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails + # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has: + # the serial console log for the current boot shows both `uosserver.service` and + # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used. + # UOS is installed here; see the README for what is actually wrong (the guest lost its + # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case + # where the unit name changes in some future UOS release. + # + # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in + # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below + # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of + # which envsubst leaves untouched — verified against the live VMI, whose rendered + # userData still carries its shell variables verbatim. There is deliberately NO + # brace-form shell expansion left in this payload, so nothing here needs escaping; if + # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER + # write an indexed expansion. userData: |- #cloud-config hostname: unifi fqdn: unifi.${BASE_DOMAIN} package_update: true package_upgrade: false - packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils] + packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl] write_files: - path: /usr/local/sbin/install-uos permissions: '0755' content: | - #!/usr/bin/env bash + #!/bin/bash set -euo pipefail - if [ -z "$${UOS_URL:-}" ]; then - UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')" - fi - [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; } - t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done" + d=/run/uos; mkdir -p $d + mountpoint -q $d || mount -o ro LABEL=UOSDATA $d + p=$(tr -d '\r\n' < $d/console-password || true) + case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac + systemctl cat uosserver.service >/dev/null 2>&1 && exit 0 + s=/var/lib/uos-installed + [ -e $s ] && exit 0 + u=$(tr -d '\r\n' < $d/installer-url || true) + case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac + t=$(mktemp) + curl -fL --retry 5 --retry-delay 10 -o $t "$u" + chmod +x $t + yes y | $t install || yes y | $t + rm -f $t; date -Is > $s + - path: /etc/systemd/system/uos-install.service + content: | + [Unit] + Description=Install UniFi OS Server + Wants=network-online.target + After=network-online.target + [Service] + Type=oneshot + ExecStart=/usr/local/sbin/install-uos + StandardOutput=journal+console + [Install] + WantedBy=multi-user.target runcmd: - - [ systemctl, enable, --now, qemu-guest-agent ] - - [ systemctl, enable, --now, podman.socket ] - - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ] + - [systemctl, enable, --now, qemu-guest-agent] + - [systemctl, enable, --now, podman.socket] + - [systemctl, enable, --now, uos-install.service] diff --git a/apps/vms/unifi/README.md b/apps/vms/unifi/README.md index 398e05f..d5093e7 100644 --- a/apps/vms/unifi/README.md +++ b/apps/vms/unifi/README.md @@ -1,18 +1,48 @@ # UniFi Network controller (UniFi OS Server) — VM onboarding -> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization -> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate -> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import -> or install cannot stall the app tier. +> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.** +> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`, +> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle +> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install +> cannot stall the app tier. > -> **Status (verified 2026-08-26):** -> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable). -> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`. -> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`. -> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services. -> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster -> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy). -> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1. +> **Status (re-verified against live prod 2026-09-14):** +> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**. +> - CDI import — **Succeeded** (100%). +> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS, +> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d. +> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services. +> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below. +> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and +> `:8080` refuse connections. + +## What is actually wrong (2026-09-14) + +The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu". +That was wrong, and acting on it would have meant reinstalling over a working controller. + +Evidence, all read-only, from the live cluster: + +| Claim | How it was checked | Result | +|---|---|---| +| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** | +| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB | +| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty | +| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address | +| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated | + +So the controller is installed and its units start; nothing downstream of the guest's own network +stack can work. The interface was fine at boot (the console log reaches `network-online.target` at +~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is +UOS taking over host networking, but **that has not been proven and cannot be, from outside the +guest.** Diagnosing it needs a shell, which is the second problem below. + +**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets +no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt +nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the +ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the +guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the +new Secret is for. ## What this is @@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation | | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS | | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster | -| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) | +| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit | | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress | -| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record | +| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record | +| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk | +| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) | | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) | -The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS -serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on -this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands. +The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on +`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium +version. Browser access is via the L4 VIP. Re-add it if/when that lands. -### Cloud-init delivery — two constraints learned the hard way +### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0) -1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman - prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the - shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a - Secret deliberately. -2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote - `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml` - therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency. +1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply + of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud + userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped + payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name + is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently + validate as an unknown field. +2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes + nothing.** The NoCloud seed's `instance-id` is exactly + `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object** + (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI + product UUID. Same id → cached datasource restored, new payload never read, `write_files` and + `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken + effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way + to force a re-run. See the long comment in `30-virtualmachine.yaml`. +3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has + been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's + envsubst therefore runs over the cloud-init payload, and the escape contract in + `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified + against the live VMI's rendered userData); brace-form shell expansion must be escaped; an + indexed expansion is a hard error that fails the whole Kustomization. ### Node prerequisites this VM forced onto the cluster @@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default 3-replica class made the import fail with insufficient storage. -## UOS install method (researched) +## UOS install method (researched 2026-09-14) -- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min - 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`. -- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from - `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on - **`https://:11443`**. -- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple - independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks. +- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. + Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi** + (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong). + Podman must be at `/usr/bin/podman`. +- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed + in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://:11443`**. +- **Is the URL a short-lived signed link?** *Probably not.* The links look like + `https://fw-download.ubnt.com/data/unifi-os-server/-linux-x64--.-x64` with + **no signature query string**, and public community installers hard-code them. It is + account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps + account-derived material out of plaintext config — but do **not** build logic around expiry. +- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.** + Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the + ecosystem actually does is pipe confirmations in: the widely used community installer + (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run + either `./ install` or a bare `sudo ./`. `install-uos` therefore does + `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a + fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest** + (no URL has been supplied, and UOS is already installed here), so treat it as "established by + convergent third-party practice", not as verified-by-us. +- The previously shipped helper had two defects worth recording: it resolved the URL from + `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists + `unifi-os-server`, and it invoked the binary with **no `install` subcommand**. ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them) @@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la | 3478 | UDP | **STUN** (device ⇄ controller) | | 10001 | UDP | **L2 device discovery** | +**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only +the ports it declares, so SSH into this guest is impossible by construction. Use the serial console. + ## L4 exposure design (the hard part) The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789` or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So: - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via - Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`). - The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the - flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config. + Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}` + (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing + `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by + the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the + existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60` + / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been + decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.) - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the - declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP. -- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`) - — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43. - Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN - reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod - masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use - explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway. -- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium - LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. - `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design. + declared ports into the guest. +- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file + used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here: + Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the + leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward), + and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw + it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and + why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`. +- **Device-adoption story:** point devices' **inform host** at the VIP + (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting, + `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own + subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:** + broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so + zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices + on other VLANs must route to the VIP and be inform-pointed anyway. +- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a + per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live. -**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less -`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy -(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`. +**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to +the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and +`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the +zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the +apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13 +flip, so this hostname is current, not a dev leftover. + +## The operator's one field — installing UOS without a shell + +`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into +the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest +`uos-install.service` reads it **on every boot**: + +```bash +sops apps/vms/unifi/unifi-uos-installer.sops.yaml # fill installer-url and/or console-password +git commit && git push # merge to the branch prod's Flux tracks +kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed +kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \ + -c guest-console-log # watch the boot + the install, read-only +``` + +Design notes that matter if you change this: + +- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte + cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the + L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted** + resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext, + operator values in SOPS. +- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and + the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the + firmware-UUID bump. +- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows + a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live + guest already has UOS, this guard is the difference between "safe to merge" and "destroys the + controller on next boot". +- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under + `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and + `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the + guest mounts a disk full of ciphertext, with the cluster reporting everything healthy. ## Open questions / risks -1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed, - account-tied** URL that **rotates/expires** (cannot be committed) and is documented as - **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for - the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**. - **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or - invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL). -2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI +1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis + requires a shell, which requires `console-password` above plus one restart. First things to run + on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`, + `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its + own (the address survived the first ~34h of this boot's life), but that would hide the cause, so + capture the state before rebooting a second time if you can. +2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the + install-method section: no official silent flag exists, `yes y | install` is the + convergent community practice, and it has not been executed on our guest. Treat the first real + install as something to watch on the console log, not as a fire-and-forget. +3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI import on this cluster). Still no checksum on an http source, and the URL tracks a *moving* Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level fixes (`nonroot-devices`, `longhorn-scratch`) — see above. -3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS. -4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the - `192.168.1.53–.60` pool (`.53` is the shared Gateway). -5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block / +4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS. +5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live + prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns + (`.23`) or the node addresses (`.10/.11/.12`). +6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block / migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`. -6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the +7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access. -7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod +8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod masquerade boundary — see the L4 section. Expect explicit inform-host adoption. +9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain + `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs + `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching + the cluster's **v1.9.0**. ## Remaining bring-up -Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook. +Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it +is written for a from-scratch cluster; the prod addresses are the live ones. -1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.** +1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`). 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms` - Kustomization (not an app wave). + Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays. 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`. 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show - `192.168.1.54`. -5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key - and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from - ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**. -6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`), + `192.168.2.21`. +5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise; + `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work: + fill `installer-url`, commit, and the guest installs itself on first boot. +6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the + console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass. +7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`), complete the first-run wizard. -7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN. -8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN +8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole + split-horizon) and from in-cluster (coredns-custom). +9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN connects (green). Test guest portal / throughput if used. -9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS. -10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version. +10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS. +11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version. **Quick health check today:** ```bash kubectl -n unifi get dv,vmi,svc kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}' +# the two that would have caught the current failure: +kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}' # expect an ipAddress +kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \ + -c compute -- ip neigh # expect 10.0.2.2 REACHABLE ``` diff --git a/apps/vms/unifi/kustomization.yaml b/apps/vms/unifi/kustomization.yaml index d67b3be..a3c4e23 100644 --- a/apps/vms/unifi/kustomization.yaml +++ b/apps/vms/unifi/kustomization.yaml @@ -11,3 +11,11 @@ resources: - 30-virtualmachine.yaml - 40-networkpolicy.yaml - 50-service-l4.yaml + # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted + # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux + # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the + # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext + # template + the operator hand-off steps live in the `.example` sibling of the file below, which + # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose + # text contains an example template's filename, on the assumption it is being applied. + - unifi-uos-installer.sops.yaml diff --git a/apps/vms/unifi/unifi-uos-installer.sops.yaml b/apps/vms/unifi/unifi-uos-installer.sops.yaml new file mode 100644 index 0000000..3192dae --- /dev/null +++ b/apps/vms/unifi/unifi-uos-installer.sops.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: Secret +metadata: + name: unifi-uos-installer + namespace: unifi +type: Opaque +stringData: + installer-url: ENC[AES256_GCM,data:CwdpCy4dKbJa7A==,iv:zdKrFad7Lg2g7bM7ALLPzp/3VIjXCahZABEscpJCYF4=,tag:inlLeAKrPrt9qvr8/Y4Hiw==,type:str] + console-password: ENC[AES256_GCM,data:F+3lMUjA092RVg==,iv:h7NenRZUEJxTjzRDGCLBMHRtLl9h122wLvUvNGfmB50=,tag:Emo5y0qOUBptnXK03sgadQ==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLMHpXYVhkblBNQWJ2NlZW + WXJPSDcxUEdYakN0R1Nydkc1a2E5anJSNEV3CmM1MUpiNzZUa0dKQXNRZGpTRm9l + UnMvMTNDL3gvUG1jVFdBbUI3aU01RkkKLS0tIGh2OUNyZzFpMXI1SWFSZENPNVVJ + R2V4N3plREx5UGJ2emprTS9WRzljRXMKCWIHr3+L+H2QxocHmA5oDdjZBfvo2Z5F + 6JLNGs0M7O0X5n+IUqyM+pU+BbqHX+y2hTOSwOgCSV9Zw+ME1kUTKg== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-14T07:13:55Z" + mac: ENC[AES256_GCM,data:A2JQR++tYY9XrRmN1x24klU4xpanx0Ab64AIPGfvtHUxwtG2JLCMMAHIe64eM60FIKT5ibmnVVeuRIUoDwm5d3WpdX2gmGpzxyr9TDhEcEA2ENQGP5TydnmcmchNcu//FfXTXTgm91+WjBke06gibKEF9H+65s2YbkOtheFUjqA=,iv:tQkW7xDGMb8+35QAx+svTQDEwlvW8AQfouopeRqtNTg=,tag:fLJ58ovDbf5qy0sxBqUfPw==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/apps/vms/unifi/unifi-uos-installer.sops.yaml.example b/apps/vms/unifi/unifi-uos-installer.sops.yaml.example new file mode 100644 index 0000000..d7274f7 --- /dev/null +++ b/apps/vms/unifi/unifi-uos-installer.sops.yaml.example @@ -0,0 +1,71 @@ +# ============================================================================= +# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never +# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF. +# +# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with +# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all. +# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see +# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on +# that disk, and the in-guest `uos-install.service` reads them on EVERY boot. +# +# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the +# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it +# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the +# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a +# working controller. `console-password` IS useful today: it is the only way to get a shell. +# +# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account +# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not +# ours to redistribute. Treat it as account-derived material, not as public configuration. +# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/ +# unifi-os-server/-linux-x64--.-x64` URL carries NO signature query string, +# and public community install scripts hard-code these URLs — so it does not behave like a +# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the +# repo's "account-derived material is not config" line clean, but do NOT design around "it expires". +# +# HAND-OFF STEPS (operator) — this is the whole job: +# (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and +# copy the download link (right-click -> Copy link address). Do NOT download it here; the guest +# fetches it itself. +# (b) Open the encrypted sibling in your editor: +# sops apps/vms/unifi/unifi-uos-installer.sops.yaml +# (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a +# placeholder rather than failing. +# (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks. +# (e) Restart the VM once so the new cloud-init instance and the new disk are picked up: +# kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed +# Watch it come up WITHOUT virtctl: +# kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \ +# -o name | head -1) -c guest-console-log +# (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443. +# +# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is +# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit +# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init +# payload ITSELF changes. +# +# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms` +# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod. +# ============================================================================= +apiVersion: v1 +kind: Secret +metadata: + name: unifi-uos-installer + namespace: unifi +type: Opaque +stringData: + # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and + # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently. + # + # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL + # (this placeholder included) is treated by the guest as "not provided yet": the unit logs + # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install. + installer-url: REPLACE-ME + # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`. + # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked, + # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards + # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside + # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with + # openssl rand -base64 24 + # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name. + console-password: REPLACE-ME diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index 0e38564..15b3dca 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -122,6 +122,23 @@ patches: provider: sops secretRef: name: sops-age + # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret + # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL, + # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize + # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the + # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is + # as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only). + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-vms + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age - patch: |- apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 4ffbfb9..ec00743 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -113,6 +113,23 @@ patches: provider: sops secretRef: name: sops-age + # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret + # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL, + # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize + # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the + # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is + # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only). + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-vms + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) --- # # ############################################################################################