From 92944486d115279a18e786e52514d096d415f56f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 17:55:41 -0600 Subject: [PATCH] feat(dns): publish public names as proxied CNAMEs to the Cloudflare Tunnel Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally, L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns gains assertion (9) tying the Gateway target to the tunnel UUID, requiring cloudflare-proxied=false on every service-source publisher, and keeping the tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture opts out of proxying. Co-Authored-By: Claude Opus 5.5 --- apps/vms/unifi/50-service-l4.yaml | 5 + apps/wave-0/plex.yaml | 2 + catalog/plex.yaml | 9 +- .../chainsaw-test.yaml | 7 +- .../check.sh | 160 ++++++++++++++++-- .../dns-exposure/chainsaw-test.yaml | 6 + .../2026-11-08-dns-via-cloudflare-tunnel.md | 77 +++++++++ platform/crafty/service-l4.yaml | 5 + platform/externaldns/README.md | 45 +++-- platform/externaldns/ddns.yaml | 14 +- platform/externaldns/helmrelease.yaml | 58 ++++--- platform/gateway/README.md | 8 +- platform/gateway/gateway.yaml | 36 ++-- platform/headlamp/httproute.yaml | 7 + runbooks/cloudflare-tunnel.md | 46 +++-- runbooks/plex-remote-access.md | 20 ++- 16 files changed, 417 insertions(+), 88 deletions(-) create mode 100644 decisions/2026-11-08-dns-via-cloudflare-tunnel.md diff --git a/apps/vms/unifi/50-service-l4.yaml b/apps/vms/unifi/50-service-l4.yaml index 18c950b..307d017 100644 --- a/apps/vms/unifi/50-service-l4.yaml +++ b/apps/vms/unifi/50-service-l4.yaml @@ -100,6 +100,11 @@ metadata: # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP. # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443. external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}" + # DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the + # gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an + # orange-clouded A for these L4 ports would be broken. Required by the L0 check + # internal-services-publish-no-public-dns (9b) on every `service`-source publisher. + external-dns.alpha.kubernetes.io/cloudflare-proxied: "false" spec: type: LoadBalancer externalTrafficPolicy: Cluster diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index 871fa90..80ff350 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -302,6 +302,8 @@ metadata: namespace: plex labels: app.kubernetes.io/name: plex + annotations: + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/catalog/plex.yaml b/catalog/plex.yaml index 50a67c6..f546d7d 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -347,7 +347,14 @@ lifecycle: # `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took. hardware: gpu: intel-quicksync -exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET) +# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN} +# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so +# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the +# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT +# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP +# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN +# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md. +exposure: internal # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET) egress: internet # normal WAN egress (docs/03; R-NET-05) storage: - name: config diff --git a/conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml b/conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml index 6d82639..34d55f8 100644 --- a/conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml +++ b/conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml @@ -92,9 +92,10 @@ # the NAME axis only. R-NET-04 / the edge port-forward posture owns the other. # - Anything about the seven non-catalog publishers' own correctness. The register records WHAT # each publishes so the claim is reviewable, which is all a static check can offer. -# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP -# rather than a routable address; that is an open operator decision recorded in -# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess. +# - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the +# Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4 +# `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare +# then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one. # - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). # ============================================================================= apiVersion: chainsaw.kyverno.io/v1alpha1 diff --git a/conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh b/conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh index 065ba82..5dd33f2 100755 --- a/conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh +++ b/conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh @@ -29,7 +29,17 @@ # ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears); # (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the # prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after -# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime. +# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime; +# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel: +# (a) the shared Gateway's `target` is EXACTLY `.cfargotunnel.com`, the UUID read from +# platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`; +# (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay +# DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead); +# (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route +# names it, and the only `service`-source publishers of one are the declared L4 DNS-only +# names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the +# tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart; +# (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear. # # WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free): # - That the record is gone from Cloudflare. Live read; belongs beside @@ -39,8 +49,11 @@ # both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate. # - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record. # This check owns the NAME axis only. -# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the -# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml. +# - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9) +# proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1` +# returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller +# honours the prefix — a future chart bump that drops it would publish PROXIED A records at the +# Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent. # - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). # # Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns @@ -141,9 +154,12 @@ def docs_of(p: Path): # unadjudicated set cannot grow without a review event. Same ratchet reasoning as # KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve. # -# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is -# the honest state after this change, and it is deliberate: converting them is an operator -# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change, +# WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a +# PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin +# address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d). +# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record — +# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each +# remaining name should be public at all stays an operator decision per name, not a renderer change, # and a cluster-wide ExternalDNS opt-in filter — the available hardening, see # platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set. # @@ -151,27 +167,28 @@ def docs_of(p: Path): # ============================================================================================= KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = { "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy": - "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a " + "publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a " "platform component, not a catalog service; it must be reachable by every browser that " "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to " "whether the five public services are — an operator decision, not a renderer one.", "platform/keycloak/httproute.yaml:keycloak/keycloak": - "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually " + "publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually " "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not " "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate " "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name " "is meant to be public either way; it is listed because the PUBLISHING object is the " "hand-authored platform route, which carries no exposure field.", "platform/monitoring/grafana-httproute.yaml:monitoring/grafana": - "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost " - "certainly should not be publicly resolvable, but it is not a catalog service and " - "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six " - "internal services (Pi-hole must answer first).", + "publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard " + "behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per " + "name (it stayed public in the 2026-11-08 tunnel decision).", "platform/headlamp/httproute.yaml:headlamp/headlamp": - "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native " - "Keycloak OIDC; same adjudication as grafana.", + "publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator " + "decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI " + "with native Keycloak OIDC; listed because it is a hand-authored route, and pinned " + "disowned by (9d).", "platform/homepage/httproute.yaml:homepage/homepage": - "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex " + "publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex " "listener. The apex is not a per-service name and cannot be withdrawn by an exposure " "declaration.", "runbooks/serving/httproute.yaml:runbooks/runbooks": @@ -180,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = { "entry names the homepage route). Deliberately un-authenticated — see the manifest's own " "header for why gating runbooks on Keycloak is a loop with no exit.", "platform/crafty/httproute.yaml:crafty/crafty": - "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an " + "publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an " "oauth2-proxy sidecar). Platform component, no exposure field.", "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp": "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> " "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS " "Gateway cannot express. A game server is the one case where a public name may be " - "genuinely wanted; the address it answers with is still RFC1918.", + "genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).", "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp": "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} " - "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored " + "(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored " "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is " "vm-demo, which declares no ingressRoutes and publishes nothing.", } @@ -515,6 +532,113 @@ check(not bad_render, f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller " f"reads {ANNOTATION_PREFIX!r}") +# ============================================================================================= +# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel. +# ============================================================================================= +CF_CONFIG = ROOT / "platform/cloudflared/config.yaml" +GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml" +TARGET_KEY = ANNOTATION_PREFIX + "target" +PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied" +UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$") + +# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the +# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel +# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c). +L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"} +# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so +# deleting one from config.yaml cannot quietly also release it from (9c). +OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"} + +cf = next(iter(docs_of(CF_CONFIG)), None) +check(cf is not None and isinstance(cf.get("ingress"), list), + f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what " + f"the tunnel refuses or which tunnel the Gateway must target") +cf = cf or {} +tunnel = str(cf.get("tunnel") or "") +check(bool(UUID_RE.match(tunnel)), + f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — " + f"the Gateway target is derived from it") +gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway" + and ident(d) == ("gateway", "shared")), None) +check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared") +want_target = f"{tunnel}.cfargotunnel.com" +got_target = annotations_of(gw or {}).get(TARGET_KEY) +check(got_target == want_target, + f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY " + f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes " + f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or " + f"an origin address of ours.") +check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])] + or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"), + "(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only " + "CNAME to .cfargotunnel.com resolves to nothing routable, so every public name is dead") + +for rel, d in dns_services: + if is_disowned(d): + continue + if HOSTNAME_KEY not in annotations_of(d): + continue + v = annotations_of(d).get(PROXIED_KEY) + check(str(v).lower() == "false", + f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries " + f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become " + f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP " + f"is RFC1918). Every `service`-source publisher must opt out explicitly.") +for rel, d in routes: + v = annotations_of(d).get(PROXIED_KEY) + check(v is None or str(v).lower() == "true", + f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced " + f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead " + f"(`{CONTROLLER_KEY}: none`) if the name should not be public.") + +refused = {str(r.get("hostname")) for r in cf.get("ingress") or [] + if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"} +check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the " + "operator's LAN/VPN-only list is known to be non-empty)") +published_l4: set[str] = set() +for rel, d in routes: + hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []} + leaked = sorted(hosts & refused) + if is_disowned(d): + unrefused = sorted(hosts - refused) + check(not unrefused, + f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the " + f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above " + f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.") + continue + check(not leaked, + f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to " + f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves " + f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.") +for rel, d in dns_services: + if is_disowned(d): + continue + hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",") + if h.strip()} + for h in sorted(hosts & refused): + published_l4.add(h) + check(h in L4_DNS_ONLY_NAMES, + f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it " + f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}") +check(L4_DNS_ONLY_NAMES <= refused, + f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a " + f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404") +check(published_l4 == L4_DNS_ONLY_NAMES, + f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer " + f"published by any `service`-source publisher — delete them from the list") + +check(OPERATOR_LAN_ONLY_NAMES <= refused, + f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing " + f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)") +for name in sorted(OPERATOR_LAN_ONLY_NAMES): + naming = [(rel, d) for rel, d in routes + if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}] + check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry") + for rel, d in naming: + check(is_disowned(d), + f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not " + f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)") + # ============================================================================================= if FAILURES: for f in FAILURES: diff --git a/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml index 244d542..e3c3de8 100644 --- a/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml +++ b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml @@ -64,6 +64,12 @@ spec: annotations: # conformance-dns. — never a literal apex. external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain])) + # DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the + # gateway names point at the Cloudflare Tunnel). Without this opt-out the probe + # record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP, + # and the exact-IP poll below can never match. Same opt-out every real + # `service`-source publisher carries (unifi, crafty-mc). + external-dns.alpha.kubernetes.io/cloudflare-proxied: "false" spec: type: LoadBalancer # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and diff --git a/decisions/2026-11-08-dns-via-cloudflare-tunnel.md b/decisions/2026-11-08-dns-via-cloudflare-tunnel.md new file mode 100644 index 0000000..ce09727 --- /dev/null +++ b/decisions/2026-11-08-dns-via-cloudflare-tunnel.md @@ -0,0 +1,77 @@ +--- +date: 2026-11-08 +branch: feat/dns-via-cloudflare-tunnel +title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard +--- + +## What changed + +One atomic change, so the conformance guard lands with the DNS flip. + +- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target` + is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was + `wan.${BASE_DOMAIN}`). Comment block rewritten. +- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because + v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The + public-target comment block rewritten, including the prefix-regression risk. +- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`: + `external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They + are the only `service`-source publishers in the tree and live (measured). +- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml` + re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The + render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none` + on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files + were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched. +- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`. +- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and + register prose updated; new assertion (9): + (9a) Gateway target == `.cfargotunnel.com`, and + `--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied: + "false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the + tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set + {unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept + consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose + updated. +- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets + `cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails. +- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`, + `platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md` + (Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation + now also swaps the Gateway UUID; rollback path). + +## What I verified + +- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from + `gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations; + `annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the + one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a + parsed `"false"` annotation overrides it. +- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation + confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails. +- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target + back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex + golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c). + `cloudflared-ingress-rules` still passes; `render.py --selftest` PASS. +- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry + `controller: none`; plex and headlamp do not (yet); the only Services with an external-dns + hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently + `wan.home.keithjackson.dev`. +- Toolbox gate result: see the PR description (rc and skipped stages quoted there). + +## What I did NOT verify + +- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs + to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched. + Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`. +- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs` + is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it. +- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through + Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate, + but Cloudflare bot/WAF settings on that path are untested. +- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class), + `target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal + names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it + cannot catch a controller that ignores the flag. No runtime alert exists for it. +- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR. +- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges; + `platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME. diff --git a/platform/crafty/service-l4.yaml b/platform/crafty/service-l4.yaml index f99fa53..5461361 100644 --- a/platform/crafty/service-l4.yaml +++ b/platform/crafty/service-l4.yaml @@ -101,6 +101,11 @@ metadata: lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}" # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target). external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}" + # DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the + # gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an + # orange-clouded A for these L4 ports would be broken. Required by the L0 check + # internal-services-publish-no-public-dns (9b) on every `service`-source publisher. + external-dns.alpha.kubernetes.io/cloudflare-proxied: "false" spec: type: LoadBalancer externalTrafficPolicy: Cluster diff --git a/platform/externaldns/README.md b/platform/externaldns/README.md index 786400b..794c62f 100644 --- a/platform/externaldns/README.md +++ b/platform/externaldns/README.md @@ -20,29 +20,44 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `.home.keit its own record. **`internal` services need NO public record** — PiHole resolves them to the internal Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only. -## Public target + in-cluster DDNS (2026-10-03) +## Public target: the Cloudflare Tunnel, proxied (2026-11-08) -Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP: +Gateway-published records are **proxied CNAMEs to the Cloudflare Tunnel** +(`platform/cloudflared`), not the WAN IP and not the Gateway VIP: ``` -plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation) -wan.home.keithjackson.dev A <- cloudflare-ddns (ddns.yaml), every 5m +gitea.home.keithjackson.dev CNAME (proxied) fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com <- external-dns +unifi.home.keithjackson.dev A (DNS-only) 192.168.2.21 <- external-dns, `service` source (L4) +crafty-mc.home... A (DNS-only) 192.168.2.22 <- external-dns, `service` source (L4) +wan.home.keithjackson.dev A (DNS-only) <- cloudflare-ddns (ddns.yaml): WireGuard udp/51820 ONLY ``` -- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it - applies to every route that publishes at all. `internal` routes (`controller: none`) still publish - nothing. -- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`. - It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry - `egress-class: vpn`. +- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`); external-dns + reads it from the Gateway only, so it applies to every route that publishes at all. Its UUID must + equal `tunnel:` in `platform/cloudflared/config.yaml` (L0 check + `internal-services-publish-no-public-dns` (9a)). +- `--cloudflare-proxied` (HelmRelease `extraArgs`) orange-clouds every record by default — the only + practical lever, because v0.23.0 reads the per-object `cloudflare-proxied` annotation from the + route/Service, never the Gateway. **Every `service`-source publisher must carry + `external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"`** (L4 ports cannot ride the HTTP + proxy); (9b) fails the gate otherwise. +- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that + set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d). +- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address. - LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS (split-horizon) and never use the public path. -- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's - OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on - the router hairpinning (NAT reflection) WAN:443 back to the Gateway. +- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's + OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS + fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The + router NAT hairpin is no longer on that path. -Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP; -`kubectl -n external-dns logs deploy/cloudflare-ddns`. +Check: `dig +short @1.1.1.1 gitea.home.keithjackson.dev` → Cloudflare anycast addresses (no CNAME is +visible for a proxied record); `kubectl -n external-dns logs deploy/external-dns`. + +### History + +2026-10-03 → 2026-11-08: records were DNS-only CNAMEs to `wan.home.keithjackson.dev`, with the router +forwarding WAN tcp/80+443 to the Gateway VIP. Before 2026-10-03 they answered the Gateway VIP itself. ## Secrets diff --git a/platform/externaldns/ddns.yaml b/platform/externaldns/ddns.yaml index 4708410..87a6d9f 100644 --- a/platform/externaldns/ddns.yaml +++ b/platform/externaldns/ddns.yaml @@ -2,9 +2,15 @@ # In-cluster dynamic DNS for the home WAN (2026-10-03). # # Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's -# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to -# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change -# moves all of them through this one record, with no Git change and no external-dns involvement. +# current public IP. +# +# WIREGUARD-ONLY SINCE 2026-11-08. From 2026-10-03 every public name external-dns published from the +# shared Gateway was a CNAME to this record. They are now proxied CNAMEs to the Cloudflare Tunnel +# (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so NOTHING in the zone +# points here any more. The record stays for exactly one consumer: the WireGuard VPN endpoint +# (udp/51820, platform/vpn-ingress, infra/vpn-ingress/new-peer.sh), which cannot ride Cloudflare's +# HTTP proxy and needs the real WAN address. It must stay DNS-only (`PROXIED` unset/false below) — +# an orange-clouded A record would hand peers a Cloudflare anycast address that drops UDP. # # OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so # external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it. @@ -64,7 +70,7 @@ spec: - name: IP6_PROVIDER value: "none" # IPv4 WAN only; leaves any AAAA alone - name: PROXIED - value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway + value: "false" # DNS-only: WireGuard udp/51820 needs the real WAN IP - name: UPDATE_CRON value: "@every 5m" - name: DELETE_ON_STOP diff --git a/platform/externaldns/helmrelease.yaml b/platform/externaldns/helmrelease.yaml index 195f81d..1362c9f 100644 --- a/platform/externaldns/helmrelease.yaml +++ b/platform/externaldns/helmrelease.yaml @@ -107,29 +107,47 @@ spec: # what this instance can see or own; per-resource exclusion only ever REMOVES an object from # this instance's view, so it cannot make prod capable of deleting a record it does not own. # --------------------------------------------------------------------------------------------- - # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public + # THE PUBLIC TARGET — THE CLOUDFLARE TUNNEL, PROXIED (2026-11-08; supersedes 2026-10-03) # - # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20, - # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the - # fix, and both still hold: - # * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY - # from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation( - # gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the - # shared Gateway therefore moves EVERY gateway-sourced record at once. - # * The home WAN IP is residential and can change, so a literal IP in Git would rot. + # History: until 2026-10-03 every gateway-sourced record answered the Gateway VIP (RFC1918); + # from 2026-10-03 they were CNAMEs to `wan.${BASE_DOMAIN}` (DNS-only, the home WAN IP, with the + # router forwarding tcp/80+443 to the VIP). Both published an origin address of ours. Now: + # * platform/gateway/gateway.yaml sets `target: .cfargotunnel.com` on the shared + # Gateway. `target` is read in v0.23.0's gateway-httproute source ONLY from the GATEWAY + # (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(gw.Annotations)`), + # never from an HTTPRoute, so this one value moves every gateway-sourced record at once. + # * `--cloudflare-proxied` below orange-clouds every record by default. It is GLOBAL because + # that is the only practical lever: v0.23.0 reads the per-object `cloudflare-proxied` + # annotation from the ROUTE (or Service), never the Gateway (source/gateway.go: + # `annotations.ProviderSpecificAnnotations(annots)` with `annots := meta.Annotations` of the + # route), so a Gateway-level opt-in does not exist. A CNAME to cfargotunnel.com MUST be + # proxied — DNS-only it resolves to nothing routable — so "proxied by default" is the right + # default for this zone's gateway names. + # * The provider honours a per-object `"false"` over the global default (provider/cloudflare + # shouldBeProxied: ParseBool of the annotation overrides proxiedByDefault). The key it reads + # is `cloudflare-proxied` (annotations.SetAnnotationPrefix re-derives + # CloudflarePrefix), i.e. `external-dns.alpha.kubernetes.io/cloudflare-proxied` here. + # + # THE `service` SOURCE MUST OPT OUT. The L4 LoadBalancers (crafty-mc, unifi) publish an A record + # at their own LB VIP; Cloudflare's proxy is HTTP(S)-only, so a proxied A for a game or UniFi + # inform port is broken by construction. Each carries `cloudflare-proxied: "false"` and stays + # DNS-only, exactly as before this change. conformance/tests/l0-static/ + # internal-services-publish-no-public-dns (assertion 9b) FAILS on any `service`-source publisher + # without it, so a new L4 Service cannot silently become a proxied record. # - # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin - # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy; - # (b) the address is kept current IN-CLUSTER. Implementation: - # * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so - # each published name becomes a CNAME to that one hostname; - # * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A - # record at the current WAN IP. That record has no external-dns TXT registry entry, so this - # instance never owns or garbage-collects it. # Internal services are unchanged: their routes carry `controller: none` (above), which is - # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4 - # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its - # LB VIP. Whether those should go public is a separate, still-open question. + # checked BEFORE the target is read, so they still publish nothing. Plex's web UI and Headlamp + # joined that set on 2026-11-08 (operator: LAN/VPN-only). `wan.${BASE_DOMAIN}` (ddns.yaml) has no + # external-dns TXT registry entry and is untouched; it now serves only WireGuard udp/51820. + # + # THE RISK THIS ADDS, AND WHAT GUARDS IT. If the annotation prefix ever stops being honoured + # again (the 2026-11-07 v0.23.0 incident), `target` is ignored and every name falls back to the + # Gateway's own address — and with this flag it would be published as a PROXIED A record at + # ${GATEWAY_VIP} (RFC1918; Cloudflare 1003/522s for every public name) while `controller: none` + # is also ignored and internal names publish. Assertion (8) of the L0 check above ties the + # prefix this tree writes to `values.annotationPrefix`; (8b) forbids overriding it here. + extraArgs: + - --cloudflare-proxied # --------------------------------------------------------------------------------------------- # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below. # diff --git a/platform/gateway/README.md b/platform/gateway/README.md index d403257..be3b837 100644 --- a/platform/gateway/README.md +++ b/platform/gateway/README.md @@ -30,5 +30,9 @@ east-west default-deny (docs/09). - **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09); add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS. -- **Public exposure** is a **manual edge port-forward** of 80/443 to the Gateway VIP (single SOHO - router, docs/09) — a documented manual step, the analogue of the old pfSense exception. +- **Public exposure** is via the **Cloudflare Tunnel** (2026-11-08, `platform/cloudflared`): the + Gateway's `external-dns.alpha.kubernetes.io/target` is `.cfargotunnel.com`, so every + published name is a proxied CNAME to the tunnel and cloudflared forwards to this Gateway's + ClusterIP. No inbound WAN 80/443 forward is needed (the old manual edge port-forward is retired — + runbooks/cloudflare-tunnel.md). LAN/VPN-only names (internal apps, plex, headlamp) carry + `controller: none` on their route and are 404'd by the tunnel. diff --git a/platform/gateway/gateway.yaml b/platform/gateway/gateway.yaml index 391a2bc..9c87108 100644 --- a/platform/gateway/gateway.yaml +++ b/platform/gateway/gateway.yaml @@ -8,22 +8,34 @@ metadata: name: shared namespace: gateway annotations: - # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every - # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is - # unreachable from the internet. With it, every published `.${BASE_DOMAIN}` (and the apex) - # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps - # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}. + # PUBLIC DNS TARGET = THE CLOUDFLARE TUNNEL (2026-11-08, operator decision; replaces the + # 2026-10-03 `wan.${BASE_DOMAIN}` target). external-dns v0.23.0's gateway-httproute source reads + # `target` from the GATEWAY only (source/gateway.go: `overrides: annotations. + # TargetsFromTargetAnnotation(gw.Annotations)`) — never from an HTTPRoute — so this one value moves + # EVERY published route on this Gateway at once. Each published `.${BASE_DOMAIN}` (and the + # apex) becomes a CNAME to `.cfargotunnel.com`, and platform/externaldns/helmrelease.yaml + # sets `--cloudflare-proxied` so the CNAME is orange-clouded: public clients resolve Cloudflare + # anycast, Cloudflare terminates edge TLS (Total TLS) and hands the request to the in-cluster + # cloudflared connectors (platform/cloudflared), which forward to this Gateway's ClusterIP. No + # origin address of ours appears in public DNS, and no inbound WAN port is needed for HTTP(S). # - # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the - # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five - # `exposure: public` apps. `internal` services stay unpublished via their - # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks - # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never - # from an HTTPRoute — see platform/externaldns/helmrelease.yaml. + # THE UUID IS A LITERAL, AND IT MUST EQUAL `tunnel:` IN platform/cloudflared/config.yaml. A CNAME + # to any other .cfargotunnel.com is a dead name (Cloudflare answers 1016). That equality is + # asserted on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns + # (assertion 9a). A literal rather than a cluster-var keeps this change off clusters/**. + # + # Scope: every route on this Gateway that publishes at all. `internal` services (and the + # operator-designated LAN/VPN-only names: plex, headlamp) stay unpublished via their + # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns + # applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's + # ingress rules, so even a hand-made record cannot reach them through the tunnel. # # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path. - external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}" + # + # `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here; + # it stays only as the WireGuard udp/51820 endpoint. + external-dns.alpha.kubernetes.io/target: "fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com" spec: gatewayClassName: cilium listeners: diff --git a/platform/headlamp/httproute.yaml b/platform/headlamp/httproute.yaml index eaccb53..cbb8c35 100644 --- a/platform/headlamp/httproute.yaml +++ b/platform/headlamp/httproute.yaml @@ -18,6 +18,13 @@ metadata: annotations: homelab.keithjackson.dev/auth-mode: oidc homelab.keithjackson.dev/auth-groups: platform-admins + # LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public + # headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep + # resolving the name via PiHole's split-horizon wildcard (-> the Gateway VIP). The Cloudflare + # Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml). + # Same mechanism the renderer emits for `exposure: internal` services; asserted by + # conformance/tests/l0-static/internal-services-publish-no-public-dns (9c). + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/runbooks/cloudflare-tunnel.md b/runbooks/cloudflare-tunnel.md index 4e6bb3e..d429e22 100644 --- a/runbooks/cloudflare-tunnel.md +++ b/runbooks/cloudflare-tunnel.md @@ -31,10 +31,29 @@ The operator decisions it encodes (2026-11-08): policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check `cloudflared-ingress-rules` fails the gate if that ever changes. -**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a -tunnel only when its record is a CNAME to `.cfargotunnel.com`. Phase 1 (this component) -creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts -firing means "the tunnel is not ready", not "users are down". +**Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to +`.cfargotunnel.com`.** Phase 1 created no such record. + +**Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes +every gateway name as a proxied CNAME to the tunnel: the shared Gateway's +`external-dns.alpha.kubernetes.io/target` is `.cfargotunnel.com` and the HelmRelease sets +`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are +disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only +A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check +`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's +UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public +users are down** — every public HTTP(S) name depends on the connectors. + +### After the Phase 2 merge — operator step (router) + +Once `dig +short @1.1.1.1 ` returns Cloudflare anycast addresses for the public names +(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more: + +1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP. +2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400** + (Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`). +3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and + `curl -m5 https:///` now times out instead of answering. Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and `CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization @@ -103,8 +122,8 @@ whole of what this Deployment needs; no `cert.pem` ever belongs in the pod. - **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one. **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`); a record created behind its back either fights it or is deleted by it, and a real hostname - routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2 - change. The only record this runbook creates is the hand-made canary in step 6, and you delete it. + routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns + (Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it. - Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2. @@ -300,6 +319,9 @@ is deleted, which is why the last step is not optional. ``` OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml) sed -i "s/^tunnel: ${OLD}$/tunnel: /" platform/cloudflared/config.yaml + # Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit; + # the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree. + sed -i "s/${OLD}\.cfargotunnel\.com/.cfargotunnel.com/" platform/gateway/gateway.yaml ``` 3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never @@ -315,8 +337,9 @@ is deleted, which is why the last step is not optional. cf tunnel delete cf tunnel list # the old tunnel is gone ``` - After the Phase 2 DNS flip, records point at `.cfargotunnel.com`: confirm external-dns - has repointed every record to the NEW UUID before deleting, or those hostnames go dark. + Since Phase 2, records point at `.cfargotunnel.com`: confirm external-dns has repointed + every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those + hostnames go dark. Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk. ## Rollback @@ -332,6 +355,9 @@ is deleted, which is why the last step is not optional. ``` Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then `flux resume kustomization cloudflared`. -- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records - (that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it +- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel` + (Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then + rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards + are **re-opened** first (they are closed after Phase 2). Never edit the records by hand: + external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked. diff --git a/runbooks/plex-remote-access.md b/runbooks/plex-remote-access.md index dad33e7..25a933d 100644 --- a/runbooks/plex-remote-access.md +++ b/runbooks/plex-remote-access.md @@ -11,6 +11,19 @@ env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway UR **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex procedures with a manual last mile. +## The Plex web UI is LAN/VPN-only (2026-11-08) + +When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the +operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the +`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record +is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI +at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer. + +**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands +clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the +WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after +the tunnel cut-over. Remote users use app.plex.tv, not our hostname. + ## What was actually broken (measured 2026-10-03) Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote @@ -44,9 +57,10 @@ either way — this repo has no such vantage point and must not pretend otherwis router by hand, so it must never move on a reschedule. While in the router, **check what else is bound to the WAN interface.** If remote management is -listening on 443 (see the certificate measurement above), move it or disable it — otherwise the -Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even -though remote *streaming* works. These are two separate paths and they fail independently. +listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this +used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since +2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the +80/443 forwards are retired; only tcp/32400 matters here.) Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous: -- 2.54.0