Operator set console-password in the SOPS secret (installer-url deliberately
left as REPLACE-ME, which install-uos handles by printing 'not set yet' and
exiting 0; the uosserver guard exits before it is even read).
Root cause of the lost address, from evidence rather than inference. KubeVirt
generates a fresh MAC per VMI when none is pinned. cloud-init writes
/etc/netplan/50-cloud-init.yaml matching the MAC it saw, and will not rewrite
it while the NoCloud instance-id is unchanged — and that instance-id is
domain.firmware.uuid, stable across restarts by design. So every VMI
recreation handed the guest a MAC its retained netplan did not match, leaving
enp1s0 administratively DOWN, never even attempting DHCP.
What the 2026-09-12 boot log shows, and why it fits: cloud-init's own device
table reads 'enp1s0 | Up: False | Address: .'; there are no apt failures
because the per-instance modules were skipped; yet UOS is installed (6.3 GB of
40 GB) and qemu-guest-agent answers — both of which required working network on
an EARLIER VMI. The MAC is identical between the current VMI status and that
boot log, so it did not change on this boot; the mismatch is against a netplan
written under a previous incarnation.
Pinned to the MAC the current VMI already holds, so nothing changes for this
guest. Combined with the instance-id bump already merged, the next restart
re-runs cloud-init, rewrites netplan against a MAC that is now permanent, and
the interface should come up — and the console password is in place as the
fallback if it does not.
Still unverified from outside: the netplan file's actual contents. That needs
the console the operator can now reach.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every other *.sops.yaml in this repo carries sops metadata and ENC[] values;
this one shipped with neither and both fields as literal REPLACE-ME. Two
consequences: 'sops <file>' fails with 'sops metadata not found', so the
documented operator step could not work; and Flux would pass the file through
undecrypted and create the Secret with the literal placeholder strings.
Encrypted in place with the repo's age recipient, placeholders intact.
Round-trip verified: sops -d returns installer-url and console-password as
REPLACE-ME, so the operator's workflow is now the normal one — sops the file,
set both values, commit.
Both fields need setting, not just the password: installer-url is also
REPLACE-ME, which I had omitted from the operator instructions.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.
Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
* inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
rejected 2049; the payload here is 1454.
* `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
disk + third disk were accepted by a server-side dry-run of the real manifest.
* the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.
secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.
CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.
There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.
README corrections (it doubles as the rebuild runbook and was misdirecting):
* VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
* "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
* "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
contract is in force.
* "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
over (an unverified firmware API, and no `install` subcommand).
* "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
* externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
* 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
* userDataSecretRef -> the YAML key is `secretRef`.
* 90-httproute-optional.yaml referenced as if it existed; it was dropped.
Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.
The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.
Co-Authored-By: Claude Opus 5 <[email protected]>
jakman85
