Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
3f980d2a17f0bb481cce44183b5d077828a6f8c2
home-network-k3s-impl/apps/vms/unifi
T
History
jakman85andClaude Opus 5 3f980d2a17 feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02) 
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).

New cluster-vars knobs (base default = dev literal, per the file's existing convention):
  K3S_API_HOST          platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
  CILIUM_LB_POOL_START  platform-config/cilium/l2-ip-pool.yaml
  CILIUM_LB_POOL_STOP   "
  UNIFI_L4_VIP          apps/vms/unifi/50-service-l4.yaml
  CRAFTY_L4_VIP         platform/crafty/service-l4.yaml
  S3_BACKUP_ENDPOINT    platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
                        semaphore}-db.yaml, and platform/velero/helmrelease.yaml

Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.

clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.

PROD ADDRESS PLAN (operator, 2026-08-31):
  nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
  Cilium LB pool .20-.30

CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.

Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.

DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.

Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-31 22:43:36 -05:00
..
10-default-deny.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
20-datavolume.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
30-virtualmachine.yaml
feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02)
2026-08-31 22:43:36 -05:00
40-networkpolicy.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
50-service-l4.yaml
feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02)
2026-08-31 22:43:36 -05:00
kustomization.yaml
fix(vms): actually apply inline userData + drop secretGenerator (amend 760084e)
2026-08-26 16:59:03 -05:00
namespace.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
README.md
docs: reconcile the documentation with the deployed reality
2026-08-27 00:00:05 -05:00

README.md

UniFi Network controller (UniFi OS Server) — VM onboarding

LIVE — partially. Reconciled by the non-gating apps-vms Flux Kustomization (clusters/base/apps-vms.yaml, dependsOn: platform-config, wait: false). VMs are a separate lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install cannot stall the app tier.

Status (verified 2026-08-26):

  • longhorn-vm StorageClass — live (RWX / Block / migratable).
  • CDI import — Succeeded (100%); unifi-rootdisk PVC Bound, 40Gi, RWX, longhorn-vm.
  • VM — Running, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on k3s-dev-1.
  • L4 LoadBalancer VIP 192.168.1.54 — allocated on both Services.
  • UniFi OS Server itself is NOT installed. Port 11443 does not answer from in-cluster (the CNP explicitly permits cluster → :11443, so this is "nothing listening", not policy). The guest is a bare, prepped Ubuntu. This is the one remaining step — see Open question 1.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File Purpose
namespace.yaml unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (40Gi). First real CDI http import on the cluster
30-virtualmachine.yaml The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, minimal inline cloud-init (SSH key, qemu-guest-agent, Podman prep)
40-networkpolicy.yaml Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record
kustomization.yaml The reconciled resource list (referenced by apps/vms/kustomization.yaml)

The optional L7 HTTPRoute (90-httproute-optional.yaml) was dropped, not merely disabled: UOS serves HTTPS on :11443, so a Gateway route needs a BackendTLSPolicy that is not confirmed on this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.

Cloud-init delivery — two constraints learned the hard way

  1. KubeVirt caps inline userData at 2048 bytes. The original rich cloud-init (full Podman prep + install-uos helper) blew past it. A userDataSecretRef was tried, then reverted; the shipped config is a minimal ~1.1 KB inline userData. Keep additions small or move to a Secret deliberately.
  2. apps-vms must NOT use postBuild.substituteFrom. Flux variable substitution rewrote $-bearing text inside the cloud-init payload and corrupted it. clusters/base/apps-vms.yaml therefore carries no substituteFrom — unlike every app wave. Do not "restore" it for consistency.

Node prerequisites this VM forced onto the cluster

  • nonroot-devices: true in the k3s server config (infra/ansible/roles/k3s_server) — the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without it: blockdev: cannot open /dev/cdi-block-volume: Permission denied.
  • longhorn-scratch — a single-replica StorageClass for CDI scratch space; the default 3-replica class made the import fail with insufficient storage.

UOS install method (researched)

  • Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at /usr/bin/podman.
  • Install: download the signed installer from fw-download.ubnt.com (URL obtained from ui.com/download/software/unifi-os-server), chmod +x, run as root. UI then on https://<host>:11443.
  • Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port Proto Use
11443 TCP UOS web UI (modern default)
8443 TCP Legacy Network UI (still served)
8080 TCP Device inform / adoption
8880 / 8843 TCP Guest portal HTTP / HTTPS
6789 TCP Mobile throughput/speedtest
3478 UDP STUN (device ⇄ controller)
10001 UDP L2 device discovery

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

  • One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned to 192.168.1.54 (lbipam.cilium.io/ips). The VIP is pulled from the existing default-pool (192.168.1.53–.60) and ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config.
  • selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest. externalTrafficPolicy: Local preserves LAN client source IP.
  • Device-adoption story: point devices' inform host at the VIP (http://192.168.1.54:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
  • Reusable pattern (== the deferred minecraft :25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. minecraft is the single-TCP-port special case of this multi-port TCP+UDP design.

UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.

Open questions / risks

  1. Unattended install (HIGH) — STILL OPEN. This is the blocker. The installer is a signed, account-tied URL that rotates/expires (cannot be committed) and is documented as interactive (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for the inline userData limit, so the install-uos helper is no longer baked into the guest. Decision needed: operator runs the install by hand at bring-up (current de-facto answer), or invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
  2. CDI http import (MEDIUM) — RESOLVED in practice. The import succeeded (first http/qcow2 CDI import on this cluster). Still no checksum on an http source, and the URL tracks a moving Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level fixes (nonroot-devices, longhorn-scratch) — see above.
  3. Boot firmware (LOW) — RESOLVED. Boots as shipped; running Ubuntu 24.04.4 LTS.
  4. VIP 192.168.1.54 (LOW) — RESOLVED. Allocated and held by both Services, inside the 192.168.1.53–.60 pool (.53 is the shared Gateway).
  5. Longhorn RWX prerequisite (was BLOCKING) — RESOLVED. longhorn-vm is live (RWX / Block / migratable); the VMI reports LiveMigratable=True and StorageLiveMigratable=True.
  6. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.
  7. Broadcast adoption (LOW, by design). L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary — see the L4 section. Expect explicit inform-host adoption.

Remaining bring-up

Steps 1–4 below are done. The sequence is kept because it is also the recovery/rebuild runbook.

  1. longhorn-vm StorageClass live; 192.168.1.54 free. Done.
  2. Wire into a reconciled path — done, via apps/vms/ + the non-gating apps-vms Kustomization (not an app wave).
  3. CDI import — done: kubectl -n unifi get dv unifi-rootdisk → Succeeded.
  4. VM + VIP — done: kubectl -n unifi get vmi unifi → Running; both Services show 192.168.1.54.
  5. → NEXT: install UOS in the guest. SSH in (the inline cloud-init installs the operator's key and qemu-guest-agent) or virtctl console unifi, fetch the signed installer URL from ui.com/download/software/unifi-os-server, then chmod +x and run it as root.
  6. Verify the UI: https://unifi.home.keithjackson.dev:11443 (or https://192.168.1.54:11443), complete the first-run wizard.
  7. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.1.54 on the cluster LAN.
  8. Adopt a device: set its inform to http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
  9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
  10. (Optional) Re-add the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.

Quick health check today:

kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
Reference in new issue View Git Blame Copy Permalink