jakman85andClaude Opus 5.5
92944486d1
Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally, L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns gains assertion (9) tying the Gateway target to the tunnel UUID, requiring cloudflare-proxied=false on every service-source publisher, and keeping the tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture opts out of proxying. Co-Authored-By: Claude Opus 5.5 <[email protected]>
139 lines
8.6 KiB
YAML
139 lines
8.6 KiB
YAML
---
|
|
# LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header).
|
|
#
|
|
# ============================================================================================
|
|
# L4 EXPOSURE (the hard part) — why the shared HTTPS Gateway is NOT enough for UniFi.
|
|
# ============================================================================================
|
|
# The Gateway API `shared` Gateway is an L7 HTTPS terminator (host-routed :443 -> backend). It
|
|
# CANNOT express: raw TCP on non-HTTP ports (8080 inform, 6789), UDP at all (3478 STUN, 10001
|
|
# discovery), or the device-facing protocols that never speak HTTP. UniFi device ADOPTION rides
|
|
# exactly those L4 ports. So the controller is exposed as a type=LoadBalancer Service whose VIP
|
|
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (dev 192.168.1.53-.60 — the
|
|
# CILIUM_LB_POOL_START/STOP cluster-vars knobs) and ARP-announced on the flat LAN by the existing
|
|
# CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
|
# machinery the Gateway ingress VIP already uses (platform-config/cilium/*). No new Cilium config
|
|
# is required; these Services just consume the pool.
|
|
#
|
|
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so TCP and UDP are
|
|
# split into two Services that SHARE ONE IP via Cilium's `lbipam.cilium.io/sharing-key`. The IP is
|
|
# PINNED (lbipam.cilium.io/ips) so devices have a STABLE inform/STUN target that never moves.
|
|
# (Alternative: one mixed-protocol Service — needs the MixedProtocolLBService feature, GA on
|
|
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
|
|
#
|
|
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
|
|
# into the guest.
|
|
#
|
|
# -------------------------------------------------------------------------------------------
|
|
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
|
# -------------------------------------------------------------------------------------------
|
|
# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
|
|
# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
|
|
# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
|
|
#
|
|
# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
|
|
# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
|
|
# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
|
|
# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
|
|
# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
|
|
# announcing node that holds no backend has nowhere to send the packet and answers ICMP port
|
|
# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
|
|
# (the leases were 8 days old); the operator's controller UI and device inform path were down.
|
|
# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
|
|
# deleting the Lease forces a re-election — and that re-election is random, so it can pick a
|
|
# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
|
|
# reference implementation this file now follows.
|
|
#
|
|
# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
|
|
# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
|
|
# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
|
|
# controller has always seen the pod-side masquerade gateway as the source, never the LAN
|
|
# client. There is a SECOND translation below the Service, and it already destroyed the very
|
|
# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
|
|
# it is giving up an address the guest could not see in the first place.
|
|
#
|
|
# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
|
|
# MAC (and the device reports its own IP inside the inform payload), not the packet source,
|
|
# and guest-portal authorization runs on the gateway appliance, not on this controller.
|
|
#
|
|
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
|
|
# is what makes this survive a reschedule and a rebuild.
|
|
#
|
|
# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
|
|
# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
|
|
# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
|
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
|
# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
|
|
# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
|
|
# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
|
# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
|
|
# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
|
#
|
|
# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
|
|
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
|
|
# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
|
|
# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
|
|
# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
|
|
# fix would be a NEW restriction that could silently break a remote site's device informing over
|
|
# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
|
|
# change, made deliberately and tested against the device fleet.
|
|
#
|
|
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
|
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
|
# Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
|
|
# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
|
|
# accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
|
|
#
|
|
# REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an
|
|
# L4-only workload as a Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing
|
|
# key; publish its A record from the Service via external-dns. minecraft = one TCP port; unifi is
|
|
# the multi-port TCP+UDP generalization.
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: unifi-l4-tcp
|
|
namespace: unifi
|
|
labels:
|
|
app.kubernetes.io/name: unifi
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "unifi"
|
|
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
|
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
|
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
|
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
|
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
|
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
|
|
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
|
|
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
|
|
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
selector:
|
|
kubevirt.io/vm: unifi
|
|
ports:
|
|
- { name: inform, port: 8080, targetPort: 8080, protocol: TCP }
|
|
- { name: ui, port: 11443, targetPort: 11443, protocol: TCP }
|
|
- { name: ui-legacy, port: 8443, targetPort: 8443, protocol: TCP }
|
|
- { name: portal-http, port: 8880, targetPort: 8880, protocol: TCP }
|
|
- { name: portal-https, port: 8843, targetPort: 8843, protocol: TCP }
|
|
- { name: throughput, port: 6789, targetPort: 6789, protocol: TCP }
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: unifi-l4-udp
|
|
namespace: unifi
|
|
labels:
|
|
app.kubernetes.io/name: unifi
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "unifi" # same key + same IP => shares the VIP with -tcp
|
|
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
selector:
|
|
kubevirt.io/vm: unifi
|
|
ports:
|
|
- { name: stun, port: 3478, targetPort: 3478, protocol: UDP }
|
|
- { name: discovery, port: 10001, targetPort: 10001, protocol: UDP }
|