Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
Files
8a2be824f600e6baaeb174d9aacc2605fee27923
home-network-k3s-impl/apps/vms/unifi
T
History
jakman85andClaude Opus 5 8a2be824f6 feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE 
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.

WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.

Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.

STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.

Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.

DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.

Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.

Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.

Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.

Serves: R-ENV-02, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 01:31:07 -05:00
..
10-default-deny.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
20-datavolume.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
30-virtualmachine.yaml
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
2026-09-01 01:31:07 -05:00
40-networkpolicy.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
50-service-l4.yaml
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
2026-09-01 01:31:07 -05:00
kustomization.yaml
fix(vms): actually apply inline userData + drop secretGenerator (amend 760084e)
2026-08-26 16:59:03 -05:00
namespace.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
README.md
docs: reconcile the documentation with the deployed reality
2026-08-27 00:00:05 -05:00

README.md

UniFi Network controller (UniFi OS Server) — VM onboarding

LIVE — partially. Reconciled by the non-gating apps-vms Flux Kustomization (clusters/base/apps-vms.yaml, dependsOn: platform-config, wait: false). VMs are a separate lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install cannot stall the app tier.

Status (verified 2026-08-26):

  • longhorn-vm StorageClass — live (RWX / Block / migratable).
  • CDI import — Succeeded (100%); unifi-rootdisk PVC Bound, 40Gi, RWX, longhorn-vm.
  • VM — Running, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on k3s-dev-1.
  • L4 LoadBalancer VIP 192.168.1.54 — allocated on both Services.
  • UniFi OS Server itself is NOT installed. Port 11443 does not answer from in-cluster (the CNP explicitly permits cluster → :11443, so this is "nothing listening", not policy). The guest is a bare, prepped Ubuntu. This is the one remaining step — see Open question 1.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File Purpose
namespace.yaml unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (40Gi). First real CDI http import on the cluster
30-virtualmachine.yaml The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, minimal inline cloud-init (SSH key, qemu-guest-agent, Podman prep)
40-networkpolicy.yaml Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record
kustomization.yaml The reconciled resource list (referenced by apps/vms/kustomization.yaml)

The optional L7 HTTPRoute (90-httproute-optional.yaml) was dropped, not merely disabled: UOS serves HTTPS on :11443, so a Gateway route needs a BackendTLSPolicy that is not confirmed on this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.

Cloud-init delivery — two constraints learned the hard way

  1. KubeVirt caps inline userData at 2048 bytes. The original rich cloud-init (full Podman prep + install-uos helper) blew past it. A userDataSecretRef was tried, then reverted; the shipped config is a minimal ~1.1 KB inline userData. Keep additions small or move to a Secret deliberately.
  2. apps-vms must NOT use postBuild.substituteFrom. Flux variable substitution rewrote $-bearing text inside the cloud-init payload and corrupted it. clusters/base/apps-vms.yaml therefore carries no substituteFrom — unlike every app wave. Do not "restore" it for consistency.

Node prerequisites this VM forced onto the cluster

  • nonroot-devices: true in the k3s server config (infra/ansible/roles/k3s_server) — the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without it: blockdev: cannot open /dev/cdi-block-volume: Permission denied.
  • longhorn-scratch — a single-replica StorageClass for CDI scratch space; the default 3-replica class made the import fail with insufficient storage.

UOS install method (researched)

  • Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at /usr/bin/podman.
  • Install: download the signed installer from fw-download.ubnt.com (URL obtained from ui.com/download/software/unifi-os-server), chmod +x, run as root. UI then on https://<host>:11443.
  • Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port Proto Use
11443 TCP UOS web UI (modern default)
8443 TCP Legacy Network UI (still served)
8080 TCP Device inform / adoption
8880 / 8843 TCP Guest portal HTTP / HTTPS
6789 TCP Mobile throughput/speedtest
3478 UDP STUN (device ⇄ controller)
10001 UDP L2 device discovery

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

  • One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned to 192.168.1.54 (lbipam.cilium.io/ips). The VIP is pulled from the existing default-pool (192.168.1.53–.60) and ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config.
  • selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest. externalTrafficPolicy: Local preserves LAN client source IP.
  • Device-adoption story: point devices' inform host at the VIP (http://192.168.1.54:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
  • Reusable pattern (== the deferred minecraft :25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. minecraft is the single-TCP-port special case of this multi-port TCP+UDP design.

UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.

Open questions / risks

  1. Unattended install (HIGH) — STILL OPEN. This is the blocker. The installer is a signed, account-tied URL that rotates/expires (cannot be committed) and is documented as interactive (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for the inline userData limit, so the install-uos helper is no longer baked into the guest. Decision needed: operator runs the install by hand at bring-up (current de-facto answer), or invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
  2. CDI http import (MEDIUM) — RESOLVED in practice. The import succeeded (first http/qcow2 CDI import on this cluster). Still no checksum on an http source, and the URL tracks a moving Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level fixes (nonroot-devices, longhorn-scratch) — see above.
  3. Boot firmware (LOW) — RESOLVED. Boots as shipped; running Ubuntu 24.04.4 LTS.
  4. VIP 192.168.1.54 (LOW) — RESOLVED. Allocated and held by both Services, inside the 192.168.1.53–.60 pool (.53 is the shared Gateway).
  5. Longhorn RWX prerequisite (was BLOCKING) — RESOLVED. longhorn-vm is live (RWX / Block / migratable); the VMI reports LiveMigratable=True and StorageLiveMigratable=True.
  6. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.
  7. Broadcast adoption (LOW, by design). L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary — see the L4 section. Expect explicit inform-host adoption.

Remaining bring-up

Steps 1–4 below are done. The sequence is kept because it is also the recovery/rebuild runbook.

  1. longhorn-vm StorageClass live; 192.168.1.54 free. Done.
  2. Wire into a reconciled path — done, via apps/vms/ + the non-gating apps-vms Kustomization (not an app wave).
  3. CDI import — done: kubectl -n unifi get dv unifi-rootdisk → Succeeded.
  4. VM + VIP — done: kubectl -n unifi get vmi unifi → Running; both Services show 192.168.1.54.
  5. → NEXT: install UOS in the guest. SSH in (the inline cloud-init installs the operator's key and qemu-guest-agent) or virtctl console unifi, fetch the signed installer URL from ui.com/download/software/unifi-os-server, then chmod +x and run it as root.
  6. Verify the UI: https://unifi.home.keithjackson.dev:11443 (or https://192.168.1.54:11443), complete the first-run wizard.
  7. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.1.54 on the cluster LAN.
  8. Adopt a device: set its inform to http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
  9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
  10. (Optional) Re-add the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.

Quick health check today:

kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
Reference in new issue View Git Blame Copy Permalink