Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
b4bc8e1cb9ddbb1e51f92edd954aa0bd6d8558b6
home-network-k3s-impl/apps/vms/unifi
T
History
jakman85andClaude Opus 5 4664b41db3 fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs 
Cilium's L2 announcer elects one node per Service to answer ARP for its
LoadBalancer VIP, and that election is NOT backend-aware. Combined with
externalTrafficPolicy: Local -- which lets a node deliver only to a backend on
itself -- an elected announcer holding no backend answers ARP for the address
and then has nowhere to send the packet, so the client gets ICMP port
unreachable. It never self-heals: the lease holder keeps renewing after the
backend moves away.

Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on
k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on
k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and
worked. Confirmed causally on the announcing node with `cilium-dbg service
list`: under Local the frontend 192.168.2.22:25565 is absent from that node's
service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565
(active)`; flipped back, it vanishes again. Durability confirmed by cordoning
k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing
node re-pointed to the new backend with no intervention.

Both files also carried a comment claiming Local "pins the VIP to the node
running the pod", which is the precise thing that is false here; both are
rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation.

The client-IP cost is stated per service rather than waved away. For unifi it
is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into
the guest, so Local was never preserving a client address the controller could
see. For crafty it is real and accepted -- player identity is a Mojang/Xbox
UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every
player at once and must not be used; no IP ban or per-IP rate limit is
configured today.

No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml:
these Services never had a source restriction (their CNPs admit `world`), so
Cluster removes no protection, and adding a LAN-only filter during an outage
fix could cut off off-LAN players or a remote site's device informing.

No CiliumNetworkPolicy change is needed, and that was verified rather than
assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by
dumping the crafty endpoint's datapath policy map with the CNP unmodified
(`reserved:remote-node` was already an Allow on the game ports). Unlike
pihole, whose policy omits `cluster` and must name remote-node explicitly.

Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with
exactly these four Services named and passes after. whole-repo scope is
necessary: ci/touched-requirements.sh has no arm for apps/* or
platform/crafty/*, so the change most likely to reintroduce this would
attribute no requirement and be gated by nothing.

HONEST LIMIT -- this does NOT end the reported outage. All four ports also
have no listening backend: UniFi OS was never installed in the VM (its README
flags this as the open blocker) and /crafty/servers is empty, so no Minecraft
server exists. Probed directly from the node, bypassing the VIP entirely,
10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two
controls answer. This commit fixes a real latent defect that would have
blackholed those VIPs once the applications exist; it cannot make an absent
application answer.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-14 01:32:47 -05:00
..
10-default-deny.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
20-datavolume.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
30-virtualmachine.yaml
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
2026-09-04 22:43:54 -05:00
40-networkpolicy.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
50-service-l4.yaml
fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs
2026-09-14 01:32:47 -05:00
kustomization.yaml
fix(vms): actually apply inline userData + drop secretGenerator (amend 760084e)
2026-08-26 16:59:03 -05:00
namespace.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
README.md
docs: reconcile the documentation with the deployed reality
2026-08-27 00:00:05 -05:00

README.md

UniFi Network controller (UniFi OS Server) — VM onboarding

LIVE — partially. Reconciled by the non-gating apps-vms Flux Kustomization (clusters/base/apps-vms.yaml, dependsOn: platform-config, wait: false). VMs are a separate lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install cannot stall the app tier.

Status (verified 2026-08-26):

  • longhorn-vm StorageClass — live (RWX / Block / migratable).
  • CDI import — Succeeded (100%); unifi-rootdisk PVC Bound, 40Gi, RWX, longhorn-vm.
  • VM — Running, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on k3s-dev-1.
  • L4 LoadBalancer VIP 192.168.1.54 — allocated on both Services.
  • UniFi OS Server itself is NOT installed. Port 11443 does not answer from in-cluster (the CNP explicitly permits cluster → :11443, so this is "nothing listening", not policy). The guest is a bare, prepped Ubuntu. This is the one remaining step — see Open question 1.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File Purpose
namespace.yaml unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (40Gi). First real CDI http import on the cluster
30-virtualmachine.yaml The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, minimal inline cloud-init (SSH key, qemu-guest-agent, Podman prep)
40-networkpolicy.yaml Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record
kustomization.yaml The reconciled resource list (referenced by apps/vms/kustomization.yaml)

The optional L7 HTTPRoute (90-httproute-optional.yaml) was dropped, not merely disabled: UOS serves HTTPS on :11443, so a Gateway route needs a BackendTLSPolicy that is not confirmed on this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.

Cloud-init delivery — two constraints learned the hard way

  1. KubeVirt caps inline userData at 2048 bytes. The original rich cloud-init (full Podman prep + install-uos helper) blew past it. A userDataSecretRef was tried, then reverted; the shipped config is a minimal ~1.1 KB inline userData. Keep additions small or move to a Secret deliberately.
  2. apps-vms must NOT use postBuild.substituteFrom. Flux variable substitution rewrote $-bearing text inside the cloud-init payload and corrupted it. clusters/base/apps-vms.yaml therefore carries no substituteFrom — unlike every app wave. Do not "restore" it for consistency.

Node prerequisites this VM forced onto the cluster

  • nonroot-devices: true in the k3s server config (infra/ansible/roles/k3s_server) — the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without it: blockdev: cannot open /dev/cdi-block-volume: Permission denied.
  • longhorn-scratch — a single-replica StorageClass for CDI scratch space; the default 3-replica class made the import fail with insufficient storage.

UOS install method (researched)

  • Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at /usr/bin/podman.
  • Install: download the signed installer from fw-download.ubnt.com (URL obtained from ui.com/download/software/unifi-os-server), chmod +x, run as root. UI then on https://<host>:11443.
  • Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port Proto Use
11443 TCP UOS web UI (modern default)
8443 TCP Legacy Network UI (still served)
8080 TCP Device inform / adoption
8880 / 8843 TCP Guest portal HTTP / HTTPS
6789 TCP Mobile throughput/speedtest
3478 UDP STUN (device ⇄ controller)
10001 UDP L2 device discovery

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

  • One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned to 192.168.1.54 (lbipam.cilium.io/ips). The VIP is pulled from the existing default-pool (192.168.1.53–.60) and ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config.
  • selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest. externalTrafficPolicy: Local preserves LAN client source IP.
  • Device-adoption story: point devices' inform host at the VIP (http://192.168.1.54:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
  • Reusable pattern (== the deferred minecraft :25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. minecraft is the single-TCP-port special case of this multi-port TCP+UDP design.

UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.

Open questions / risks

  1. Unattended install (HIGH) — STILL OPEN. This is the blocker. The installer is a signed, account-tied URL that rotates/expires (cannot be committed) and is documented as interactive (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for the inline userData limit, so the install-uos helper is no longer baked into the guest. Decision needed: operator runs the install by hand at bring-up (current de-facto answer), or invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
  2. CDI http import (MEDIUM) — RESOLVED in practice. The import succeeded (first http/qcow2 CDI import on this cluster). Still no checksum on an http source, and the URL tracks a moving Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level fixes (nonroot-devices, longhorn-scratch) — see above.
  3. Boot firmware (LOW) — RESOLVED. Boots as shipped; running Ubuntu 24.04.4 LTS.
  4. VIP 192.168.1.54 (LOW) — RESOLVED. Allocated and held by both Services, inside the 192.168.1.53–.60 pool (.53 is the shared Gateway).
  5. Longhorn RWX prerequisite (was BLOCKING) — RESOLVED. longhorn-vm is live (RWX / Block / migratable); the VMI reports LiveMigratable=True and StorageLiveMigratable=True.
  6. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.
  7. Broadcast adoption (LOW, by design). L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary — see the L4 section. Expect explicit inform-host adoption.

Remaining bring-up

Steps 1–4 below are done. The sequence is kept because it is also the recovery/rebuild runbook.

  1. longhorn-vm StorageClass live; 192.168.1.54 free. Done.
  2. Wire into a reconciled path — done, via apps/vms/ + the non-gating apps-vms Kustomization (not an app wave).
  3. CDI import — done: kubectl -n unifi get dv unifi-rootdisk → Succeeded.
  4. VM + VIP — done: kubectl -n unifi get vmi unifi → Running; both Services show 192.168.1.54.
  5. → NEXT: install UOS in the guest. SSH in (the inline cloud-init installs the operator's key and qemu-guest-agent) or virtctl console unifi, fetch the signed installer URL from ui.com/download/software/unifi-os-server, then chmod +x and run it as root.
  6. Verify the UI: https://unifi.home.keithjackson.dev:11443 (or https://192.168.1.54:11443), complete the first-run wizard.
  7. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.1.54 on the cluster LAN.
  8. Adopt a device: set its inform to http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
  9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
  10. (Optional) Re-add the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.

Quick health check today:

kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
Reference in new issue View Git Blame Copy Permalink