Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
cd64b4b6726f149df81bf879a90141335ca7e1e2
home-network-k3s-impl/apps/vms/unifi
T
History
jakman85andClaude Opus 5 c8c1d2935d
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain 
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 22:43:54 -05:00
..
10-default-deny.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
20-datavolume.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
30-virtualmachine.yaml
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
2026-09-04 22:43:54 -05:00
40-networkpolicy.yaml
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
2026-08-26 23:59:02 -05:00
50-service-l4.yaml
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
2026-09-04 22:43:54 -05:00
kustomization.yaml
fix(vms): actually apply inline userData + drop secretGenerator (amend 760084e)
2026-08-26 16:59:03 -05:00
namespace.yaml
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
2026-08-26 15:05:05 -05:00
README.md
docs: reconcile the documentation with the deployed reality
2026-08-27 00:00:05 -05:00

README.md

UniFi Network controller (UniFi OS Server) — VM onboarding

LIVE — partially. Reconciled by the non-gating apps-vms Flux Kustomization (clusters/base/apps-vms.yaml, dependsOn: platform-config, wait: false). VMs are a separate lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install cannot stall the app tier.

Status (verified 2026-08-26):

  • longhorn-vm StorageClass — live (RWX / Block / migratable).
  • CDI import — Succeeded (100%); unifi-rootdisk PVC Bound, 40Gi, RWX, longhorn-vm.
  • VM — Running, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on k3s-dev-1.
  • L4 LoadBalancer VIP 192.168.1.54 — allocated on both Services.
  • UniFi OS Server itself is NOT installed. Port 11443 does not answer from in-cluster (the CNP explicitly permits cluster → :11443, so this is "nothing listening", not policy). The guest is a bare, prepped Ubuntu. This is the one remaining step — see Open question 1.

What this is

A KubeVirt VirtualMachine running UniFi OS Server (UOS) — Ubiquiti's current self-host product. The legacy unifi .deb/unifi-network-server is deprecated. UOS is a single installer that runs on Ubuntu Server and bundles UniFi OS + the Network application + its own embedded datastore inside a Podman container stack — so no separate MongoDB companion is needed (this supersedes the earlier "unifi needs a Mongo container" note).

Files

File Purpose
namespace.yaml unifi namespace: app-namespace label (Kyverno + default-deny apply), gateway-access=shared, backup annotation
10-default-deny.yaml R-NET-01 baseline: deny all, allow only DNS
20-datavolume.yaml CDI http import of the Ubuntu 24.04 cloud image onto a longhorn-vm RWX/Block disk (40Gi). First real CDI http import on the cluster
30-virtualmachine.yaml The VM: 2 vCPU / 4Gi, running: true, evictionStrategy: LiveMigrate, minimal inline cloud-init (SSH key, qemu-guest-agent, Podman prep)
40-networkpolicy.yaml Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress
50-service-l4.yaml Two type=LoadBalancer Services (TCP+UDP) sharing one pinned Cilium VIP 192.168.1.54; external-dns A record
kustomization.yaml The reconciled resource list (referenced by apps/vms/kustomization.yaml)

The optional L7 HTTPRoute (90-httproute-optional.yaml) was dropped, not merely disabled: UOS serves HTTPS on :11443, so a Gateway route needs a BackendTLSPolicy that is not confirmed on this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.

Cloud-init delivery — two constraints learned the hard way

  1. KubeVirt caps inline userData at 2048 bytes. The original rich cloud-init (full Podman prep + install-uos helper) blew past it. A userDataSecretRef was tried, then reverted; the shipped config is a minimal ~1.1 KB inline userData. Keep additions small or move to a Secret deliberately.
  2. apps-vms must NOT use postBuild.substituteFrom. Flux variable substitution rewrote $-bearing text inside the cloud-init payload and corrupted it. clusters/base/apps-vms.yaml therefore carries no substituteFrom — unlike every app wave. Do not "restore" it for consistency.

Node prerequisites this VM forced onto the cluster

  • nonroot-devices: true in the k3s server config (infra/ansible/roles/k3s_server) — the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it. Without it: blockdev: cannot open /dev/cdi-block-volume: Permission denied.
  • longhorn-scratch — a single-replica StorageClass for CDI scratch space; the default 3-replica class made the import fail with insufficient storage.

UOS install method (researched)

  • Host: Ubuntu 24.04 LTS x86-64, systemd, Podman 4.9.3+, slirp4netns 1.2+, uidmap. Min 2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at /usr/bin/podman.
  • Install: download the signed installer from fw-download.ubnt.com (URL obtained from ui.com/download/software/unifi-os-server), chmod +x, run as root. UI then on https://<host>:11443.
  • Confidence: High on requirements, ports, and the download→chmod→run shape (multiple independent 2025–2026 sources agree). Lower on fully unattended install — see risks.

Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)

Port Proto Use
11443 TCP UOS web UI (modern default)
8443 TCP Legacy Network UI (still served)
8080 TCP Device inform / adoption
8880 / 8843 TCP Guest portal HTTP / HTTPS
6789 TCP Mobile throughput/speedtest
3478 UDP STUN (device ⇄ controller)
10001 UDP L2 device discovery

L4 exposure design (the hard part)

The shared Gateway is L7 HTTPS only (:443 host-routed). It cannot carry raw TCP 8080/6789 or any UDP (3478/10001) — exactly the ports adoption rides. So:

  • One VIP, two Services. unifi-l4-tcp + unifi-l4-udp share a single LoadBalancer IP via Cilium lbipam.cilium.io/sharing-key: unifi, pinned to 192.168.1.54 (lbipam.cilium.io/ips). The VIP is pulled from the existing default-pool (192.168.1.53–.60) and ARP-announced on the flat LAN by the existing default-l2 CiliumL2AnnouncementPolicy — no new Cilium config.
  • selector: kubevirt.io/vm=unifi → the virt-launcher pod; the VM's masquerade iface forwards the declared ports into the guest. externalTrafficPolicy: Local preserves LAN client source IP.
  • Device-adoption story: point devices' inform host at the VIP (http://192.168.1.54:8080/inform) — via the controller's Override inform host setting, set-inform on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN reach it. Limitation: broadcast-based L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
  • Reusable pattern (== the deferred minecraft :25565 problem): L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns. minecraft is the single-TCP-port special case of this multi-port TCP+UDP design.

UI access: https://unifi.home.keithjackson.dev:11443 (A record → VIP). The port-less https://unifi.home.keithjackson.dev via the Gateway is optional and needs a BackendTLSPolicy (UOS is HTTPS-only on 11443) — see 90-httproute-optional.yaml.

Open questions / risks

  1. Unattended install (HIGH) — STILL OPEN. This is the blocker. The installer is a signed, account-tied URL that rotates/expires (cannot be committed) and is documented as interactive (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for the inline userData limit, so the install-uos helper is no longer baked into the guest. Decision needed: operator runs the install by hand at bring-up (current de-facto answer), or invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
  2. CDI http import (MEDIUM) — RESOLVED in practice. The import succeeded (first http/qcow2 CDI import on this cluster). Still no checksum on an http source, and the URL tracks a moving Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level fixes (nonroot-devices, longhorn-scratch) — see above.
  3. Boot firmware (LOW) — RESOLVED. Boots as shipped; running Ubuntu 24.04.4 LTS.
  4. VIP 192.168.1.54 (LOW) — RESOLVED. Allocated and held by both Services, inside the 192.168.1.53–.60 pool (.53 is the shared Gateway).
  5. Longhorn RWX prerequisite (was BLOCKING) — RESOLVED. longhorn-vm is live (RWX / Block / migratable); the VMI reports LiveMigratable=True and StorageLiveMigratable=True.
  6. Licensing (LOW). UOS self-host is free; a Ubiquiti account is needed only to fetch the installer and for optional remote access.
  7. Broadcast adoption (LOW, by design). L2 auto-discovery (10001/udp) does not cross the pod masquerade boundary — see the L4 section. Expect explicit inform-host adoption.

Remaining bring-up

Steps 1–4 below are done. The sequence is kept because it is also the recovery/rebuild runbook.

  1. longhorn-vm StorageClass live; 192.168.1.54 free. Done.
  2. Wire into a reconciled path — done, via apps/vms/ + the non-gating apps-vms Kustomization (not an app wave).
  3. CDI import — done: kubectl -n unifi get dv unifi-rootdisk → Succeeded.
  4. VM + VIP — done: kubectl -n unifi get vmi unifi → Running; both Services show 192.168.1.54.
  5. → NEXT: install UOS in the guest. SSH in (the inline cloud-init installs the operator's key and qemu-guest-agent) or virtctl console unifi, fetch the signed installer URL from ui.com/download/software/unifi-os-server, then chmod +x and run it as root.
  6. Verify the UI: https://unifi.home.keithjackson.dev:11443 (or https://192.168.1.54:11443), complete the first-run wizard.
  7. Verify DNS: dig unifi.home.keithjackson.dev → 192.168.1.54 on the cluster LAN.
  8. Adopt a device: set its inform to http://192.168.1.54:8080/inform; confirm it appears + STUN connects (green). Test guest portal / throughput if used.
  9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
  10. (Optional) Re-add the L7 HTTPRoute once BackendTLSPolicy is confirmed on this Cilium version.

Quick health check today:

kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
Reference in new issue View Git Blame Copy Permalink