diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 951a0d1..210b760 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4123,3 +4123,63 @@ so the new state would have been unprotected until 03:00. I took a one-off backu Server", 20 slots, **no whitelist**), so the server is reachable by anyone who can reach the VIP. A whitelist or `loadBalancerSourceRanges` would restrict it — but restricting could cut off off-LAN players, which is why it is not mine to decide. + +--- + +## 2026-09-14 — UniFi OS was NEVER the blocker: it is installed; the guest lost its IP + +Third correction in two days to a claim I made confidently. I told the operator UniFi OS Server had +never been installed and that installing it was their next step. **Both halves were wrong**, and I had +taken them from `apps/vms/unifi/README.md`, which was itself stale — I repeated documentation instead +of checking the machine. + +**Verified independently on live prod:** + +| claim | evidence | +|---|---| +| UOS **is** installed and running | guest console log: `Started uosserver.service — UniFi OS Server Service` and `uosserver-updater.service` | +| it is a real install, not a stub | guest filesystem `/` holds **6.3 GB of 40 GB**; bare Ubuntu plus our packages is ~2 GB | +| **the guest has NO address** | `.status.interfaces` → `enp1s0`, `linkState: up`, **no `ipAddress`**; the VMI IP column is empty | +| confirmed at L2 | `ip neigh` inside virt-launcher: `10.0.2.2 dev k6t-eth0 **FAILED**` — the guest does not answer ARP | +| VIP / policy exonerated | `/dev/tcp` from virt-launcher to the pod IP *and* to `10.0.2.2`: 11443, 8443, 8080 and **22** all RST | + +So the controller runs and nothing can reach it, because its NIC lost its DHCP address. The cause is +not determinable from outside the guest. + +**This nearly caused real damage.** Had the work proceeded on my stated premise it would have +*reinstalled UniFi OS over a working controller*. The shipped `install-uos` now bails the moment +systemd knows a `uosserver` unit — a guard that exists only because the agent checked the premise +instead of executing it. + +**Second finding: there is no way into the guest at all.** No SSH key was ever installed, the stock +`ubuntu` account is locked, and SSH cannot work by construction — KubeVirt masquerade forwards only +declared ports and 22 is not among them, which the RST above confirms. So the deliverable became +*console access*: a SOPS secret carrying `installer-url` **and `console-password`**, mounted as a +read-only NoCloud disk and re-read on every boot. + +**The discovery that makes any of this work.** NoCloud's `instance-id` is +`spec.template.spec.domain.firmware.uuid`, stamped on the VM object — cloud-init compares it against +the DMI product UUID, so **editing `userData` alone changes nothing on a second boot**: the cached +datasource is restored and `write_files`/`runcmd` are skipped entirely. That is why a previously +shipped "unattended install" could never have taken effect, and why an earlier attempt was "tried and +reverted". The UUID is now pinned in git and bumped once. (Also: the inline `userData` cap is exactly +**2048 bytes**, established by server-side dry-run — 2048 accepted, 2049 rejected — and the payload +ships at 1454, so it stayed inline.) + +Deliberately NOT done: putting the whole cloud-config behind `secretRef`. That would hide the shell in +ciphertext, beyond the reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over the +decrypted resource — exactly the nextcloud-db shape from 2026-09-13. Script plaintext, operator values +in SOPS. + +**Unattended install remains unverified BY US.** Ubiquiti documents no silent flag, answer file or +environment variables; `yes y | install` is convergent community practice and is shipped as +such, labelled third-party. No Ubiquiti credential was touched and no installer was downloaded. + +**The operator's real next step is diagnosis, not installation:** set `console-password`, restart the +VMI, then `virtctl -n unifi console unifi` and find out why `enp1s0` has no address — `ip -4 a`, +`networkctl status enp1s0`, `journalctl -u systemd-networkd`, `ls /etc/netplan`. Capture state before +restarting, since a restart may silently restore DHCP and destroy the evidence. + +The README's dev-era addressing was corrected throughout (VIP `.1.54`→`192.168.2.21`, pool and Gateway, +device subnet, node name) along with several stale factual claims, since it doubles as the rebuild +runbook.