Merge pull request 'feat(dns): public names via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only' (#156) from feat/dns-via-cloudflare-tunnel into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped

Reviewed-on: #156
Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #156.
This commit is contained in:
jakman85 committed 2026-11-09 01:27:06 +00:00
commit 020e736541
16 files changed
+417 -88

No files matched your search

+5
View File
@@ -100,6 +100,11 @@ metadata:
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP. # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443. # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}" external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec: spec:
type: LoadBalancer type: LoadBalancer
externalTrafficPolicy: Cluster externalTrafficPolicy: Cluster
+2
View File
@@ -302,6 +302,8 @@ metadata:
namespace: plex namespace: plex
labels: labels:
app.kubernetes.io/name: plex app.kubernetes.io/name: plex
annotations:
external-dns.alpha.kubernetes.io/controller: none
spec: spec:
parentRefs: parentRefs:
- name: shared - name: shared
+8 -1
View File
@@ -347,7 +347,14 @@ lifecycle:
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took. # `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
hardware: hardware:
gpu: intel-quicksync gpu: intel-quicksync
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET) # EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
exposure: internal # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05) egress: internet # normal WAN egress (docs/03; R-NET-05)
storage: storage:
- name: config - name: config
@@ -92,9 +92,10 @@
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other. # the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT # - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
# each publishes so the claim is reviewable, which is all a static check can offer. # each publishes so the claim is reviewable, which is all a static check can offer.
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP # - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
# rather than a routable address; that is an open operator decision recorded in # Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess. # `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
# then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). # - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
# ============================================================================= # =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1 apiVersion: chainsaw.kyverno.io/v1alpha1
@@ -29,7 +29,17 @@
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears); # ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
# (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the # (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
# prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after # prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime. # the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
# (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
# platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
# (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
# DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
# (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
# names it, and the only `service`-source publishers of one are the declared L4 DNS-only
# names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
# tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
# (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
# #
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free): # WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
# - That the record is gone from Cloudflare. Live read; belongs beside # - That the record is gone from Cloudflare. Live read; belongs beside
@@ -39,8 +49,11 @@
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate. # both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record. # - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
# This check owns the NAME axis only. # This check owns the NAME axis only.
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the # - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml. # proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
# returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
# honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
# Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). # - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
# #
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns # Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
@@ -141,9 +154,12 @@ def docs_of(p: Path):
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as # unadjudicated set cannot grow without a review event. Same ratchet reasoning as
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve. # KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
# #
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is # WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
# the honest state after this change, and it is deliberate: converting them is an operator # PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change, # address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
# remaining name should be public at all stays an operator decision per name, not a renderer change,
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see # and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set. # platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
# #
@@ -151,27 +167,28 @@ def docs_of(p: Path):
# ============================================================================================= # =============================================================================================
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = { KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy": "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a " "publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
"platform component, not a catalog service; it must be reachable by every browser that " "platform component, not a catalog service; it must be reachable by every browser that "
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to " "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
"whether the five public services are — an operator decision, not a renderer one.", "whether the five public services are — an operator decision, not a renderer one.",
"platform/keycloak/httproute.yaml:keycloak/keycloak": "platform/keycloak/httproute.yaml:keycloak/keycloak":
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually " "publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not " "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate " "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name " "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
"is meant to be public either way; it is listed because the PUBLISHING object is the " "is meant to be public either way; it is listed because the PUBLISHING object is the "
"hand-authored platform route, which carries no exposure field.", "hand-authored platform route, which carries no exposure field.",
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana": "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost " "publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
"certainly should not be publicly resolvable, but it is not a catalog service and " "behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six " "name (it stayed public in the 2026-11-08 tunnel decision).",
"internal services (Pi-hole must answer first).",
"platform/headlamp/httproute.yaml:headlamp/headlamp": "platform/headlamp/httproute.yaml:headlamp/headlamp":
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native " "publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
"Keycloak OIDC; same adjudication as grafana.", "decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
"with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
"disowned by (9d).",
"platform/homepage/httproute.yaml:homepage/homepage": "platform/homepage/httproute.yaml:homepage/homepage":
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex " "publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure " "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
"declaration.", "declaration.",
"runbooks/serving/httproute.yaml:runbooks/runbooks": "runbooks/serving/httproute.yaml:runbooks/runbooks":
@@ -180,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own " "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
"header for why gating runbooks on Keycloak is a loop with no exit.", "header for why gating runbooks on Keycloak is a loop with no exit.",
"platform/crafty/httproute.yaml:crafty/crafty": "platform/crafty/httproute.yaml:crafty/crafty":
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an " "publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
"oauth2-proxy sidecar). Platform component, no exposure field.", "oauth2-proxy sidecar). Platform component, no exposure field.",
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp": "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> " "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS " "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
"Gateway cannot express. A game server is the one case where a public name may be " "Gateway cannot express. A game server is the one case where a public name may be "
"genuinely wanted; the address it answers with is still RFC1918.", "genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp": "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} " "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored " "(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is " "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
"vm-demo, which declares no ingressRoutes and publishes nothing.", "vm-demo, which declares no ingressRoutes and publishes nothing.",
} }
@@ -515,6 +532,113 @@ check(not bad_render,
f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller " f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
f"reads {ANNOTATION_PREFIX!r}") f"reads {ANNOTATION_PREFIX!r}")
# =============================================================================================
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
# =============================================================================================
CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
TARGET_KEY = ANNOTATION_PREFIX + "target"
PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
# deleting one from config.yaml cannot quietly also release it from (9c).
OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
cf = next(iter(docs_of(CF_CONFIG)), None)
check(cf is not None and isinstance(cf.get("ingress"), list),
f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
f"the tunnel refuses or which tunnel the Gateway must target")
cf = cf or {}
tunnel = str(cf.get("tunnel") or "")
check(bool(UUID_RE.match(tunnel)),
f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
f"the Gateway target is derived from it")
gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
and ident(d) == ("gateway", "shared")), None)
check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
want_target = f"{tunnel}.cfargotunnel.com"
got_target = annotations_of(gw or {}).get(TARGET_KEY)
check(got_target == want_target,
f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
f"an origin address of ours.")
check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
"(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
"CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
for rel, d in dns_services:
if is_disowned(d):
continue
if HOSTNAME_KEY not in annotations_of(d):
continue
v = annotations_of(d).get(PROXIED_KEY)
check(str(v).lower() == "false",
f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
for rel, d in routes:
v = annotations_of(d).get(PROXIED_KEY)
check(v is None or str(v).lower() == "true",
f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
"operator's LAN/VPN-only list is known to be non-empty)")
published_l4: set[str] = set()
for rel, d in routes:
hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
leaked = sorted(hosts & refused)
if is_disowned(d):
unrefused = sorted(hosts - refused)
check(not unrefused,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
continue
check(not leaked,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
for rel, d in dns_services:
if is_disowned(d):
continue
hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
if h.strip()}
for h in sorted(hosts & refused):
published_l4.add(h)
check(h in L4_DNS_ONLY_NAMES,
f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
check(L4_DNS_ONLY_NAMES <= refused,
f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
check(published_l4 == L4_DNS_ONLY_NAMES,
f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
f"published by any `service`-source publisher — delete them from the list")
check(OPERATOR_LAN_ONLY_NAMES <= refused,
f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
for name in sorted(OPERATOR_LAN_ONLY_NAMES):
naming = [(rel, d) for rel, d in routes
if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
for rel, d in naming:
check(is_disowned(d),
f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
# ============================================================================================= # =============================================================================================
if FAILURES: if FAILURES:
for f in FAILURES: for f in FAILURES:
@@ -64,6 +64,12 @@ spec:
annotations: annotations:
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex. # conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain])) external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
# DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
# record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
# and the exact-IP poll below can never match. Same opt-out every real
# `service`-source publisher carries (unifi, crafty-mc).
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec: spec:
type: LoadBalancer type: LoadBalancer
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
@@ -0,0 +1,77 @@
---
date: 2026-11-08
branch: feat/dns-via-cloudflare-tunnel
title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard
---
## What changed
One atomic change, so the conformance guard lands with the DNS flip.
- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target`
is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was
`wan.${BASE_DOMAIN}`). Comment block rewritten.
- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because
v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The
public-target comment block rewritten, including the prefix-regression risk.
- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`:
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They
are the only `service`-source publishers in the tree and live (measured).
- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml`
re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The
render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none`
on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files
were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched.
- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`.
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and
register prose updated; new assertion (9):
(9a) Gateway target == `<tunnel: from platform/cloudflared/config.yaml>.cfargotunnel.com`, and
`--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied:
"false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the
tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set
{unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept
consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose
updated.
- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets
`cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails.
- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`,
`platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md`
(Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation
now also swaps the Gateway UUID; rollback path).
## What I verified
- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from
`gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations;
`annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the
one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a
parsed `"false"` annotation overrides it.
- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation
confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails.
- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target
back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex
golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c).
`cloudflared-ingress-rules` still passes; `render.py --selftest` PASS.
- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry
`controller: none`; plex and headlamp do not (yet); the only Services with an external-dns
hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently
`wan.home.keithjackson.dev`.
- Toolbox gate result: see the PR description (rc and skipped stages quoted there).
## What I did NOT verify
- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs
to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched.
Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`.
- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs`
is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it.
- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through
Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate,
but Cloudflare bot/WAF settings on that path are untested.
- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class),
`target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal
names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it
cannot catch a controller that ignores the flag. No runtime alert exists for it.
- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR.
- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges;
`platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME.
+5
View File
@@ -101,6 +101,11 @@ metadata:
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}" lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
# external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target). # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}" external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec: spec:
type: LoadBalancer type: LoadBalancer
externalTrafficPolicy: Cluster externalTrafficPolicy: Cluster
+30 -15
View File
@@ -20,29 +20,44 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only. Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
## Public target + in-cluster DDNS (2026-10-03) ## Public target: the Cloudflare Tunnel, proxied (2026-11-08)
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP: Gateway-published records are **proxied CNAMEs to the Cloudflare Tunnel**
(`platform/cloudflared`), not the WAN IP and not the Gateway VIP:
``` ```
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation) gitea.home.keithjackson.dev CNAME (proxied) fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com <- external-dns
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m unifi.home.keithjackson.dev A (DNS-only) 192.168.2.21 <- external-dns, `service` source (L4)
crafty-mc.home... A (DNS-only) 192.168.2.22 <- external-dns, `service` source (L4)
wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddns (ddns.yaml): WireGuard udp/51820 ONLY
``` ```
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it - The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`); external-dns
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish reads it from the Gateway only, so it applies to every route that publishes at all. Its UUID must
nothing. equal `tunnel:` in `platform/cloudflared/config.yaml` (L0 check
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`. `internal-services-publish-no-public-dns` (9a)).
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry - `--cloudflare-proxied` (HelmRelease `extraArgs`) orange-clouds every record by default — the only
`egress-class: vpn`. practical lever, because v0.23.0 reads the per-object `cloudflare-proxied` annotation from the
route/Service, never the Gateway. **Every `service`-source publisher must carry
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"`** (L4 ports cannot ride the HTTP
proxy); (9b) fails the gate otherwise.
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS - LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
(split-horizon) and never use the public path. (split-horizon) and never use the public path.
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's - **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
the router hairpinning (NAT reflection) WAN:443 back to the Gateway. fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
router NAT hairpin is no longer on that path.
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP; Check: `dig +short @1.1.1.1 gitea.home.keithjackson.dev` → Cloudflare anycast addresses (no CNAME is
`kubectl -n external-dns logs deploy/cloudflare-ddns`. visible for a proxied record); `kubectl -n external-dns logs deploy/external-dns`.
### History
2026-10-03 → 2026-11-08: records were DNS-only CNAMEs to `wan.home.keithjackson.dev`, with the router
forwarding WAN tcp/80+443 to the Gateway VIP. Before 2026-10-03 they answered the Gateway VIP itself.
## Secrets ## Secrets
+10 -4
View File
@@ -2,9 +2,15 @@
# In-cluster dynamic DNS for the home WAN (2026-10-03). # In-cluster dynamic DNS for the home WAN (2026-10-03).
# #
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's # Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to # current public IP.
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change #
# moves all of them through this one record, with no Git change and no external-dns involvement. # WIREGUARD-ONLY SINCE 2026-11-08. From 2026-10-03 every public name external-dns published from the
# shared Gateway was a CNAME to this record. They are now proxied CNAMEs to the Cloudflare Tunnel
# (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so NOTHING in the zone
# points here any more. The record stays for exactly one consumer: the WireGuard VPN endpoint
# (udp/51820, platform/vpn-ingress, infra/vpn-ingress/new-peer.sh), which cannot ride Cloudflare's
# HTTP proxy and needs the real WAN address. It must stay DNS-only (`PROXIED` unset/false below) —
# an orange-clouded A record would hand peers a Cloudflare anycast address that drops UDP.
# #
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so # OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it. # external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
@@ -64,7 +70,7 @@ spec:
- name: IP6_PROVIDER - name: IP6_PROVIDER
value: "none" # IPv4 WAN only; leaves any AAAA alone value: "none" # IPv4 WAN only; leaves any AAAA alone
- name: PROXIED - name: PROXIED
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway value: "false" # DNS-only: WireGuard udp/51820 needs the real WAN IP
- name: UPDATE_CRON - name: UPDATE_CRON
value: "@every 5m" value: "@every 5m"
- name: DELETE_ON_STOP - name: DELETE_ON_STOP
+38 -20
View File
@@ -107,29 +107,47 @@ spec:
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
# this instance's view, so it cannot make prod capable of deleting a record it does not own. # this instance's view, so it cannot make prod capable of deleting a record it does not own.
# --------------------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------------------
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public # THE PUBLIC TARGET — THE CLOUDFLARE TUNNEL, PROXIED (2026-11-08; supersedes 2026-10-03)
# #
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20, # History: until 2026-10-03 every gateway-sourced record answered the Gateway VIP (RFC1918);
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the # from 2026-10-03 they were CNAMEs to `wan.${BASE_DOMAIN}` (DNS-only, the home WAN IP, with the
# fix, and both still hold: # router forwarding tcp/80+443 to the VIP). Both published an origin address of ours. Now:
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY # * platform/gateway/gateway.yaml sets `target: <tunnel-uuid>.cfargotunnel.com` on the shared
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation( # Gateway. `target` is read in v0.23.0's gateway-httproute source ONLY from the GATEWAY
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the # (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(gw.Annotations)`),
# shared Gateway therefore moves EVERY gateway-sourced record at once. # never from an HTTPRoute, so this one value moves every gateway-sourced record at once.
# * The home WAN IP is residential and can change, so a literal IP in Git would rot. # * `--cloudflare-proxied` below orange-clouds every record by default. It is GLOBAL because
# that is the only practical lever: v0.23.0 reads the per-object `cloudflare-proxied`
# annotation from the ROUTE (or Service), never the Gateway (source/gateway.go:
# `annotations.ProviderSpecificAnnotations(annots)` with `annots := meta.Annotations` of the
# route), so a Gateway-level opt-in does not exist. A CNAME to cfargotunnel.com MUST be
# proxied — DNS-only it resolves to nothing routable — so "proxied by default" is the right
# default for this zone's gateway names.
# * The provider honours a per-object `"false"` over the global default (provider/cloudflare
# shouldBeProxied: ParseBool of the annotation overrides proxiedByDefault). The key it reads
# is `<annotationPrefix>cloudflare-proxied` (annotations.SetAnnotationPrefix re-derives
# CloudflarePrefix), i.e. `external-dns.alpha.kubernetes.io/cloudflare-proxied` here.
#
# THE `service` SOURCE MUST OPT OUT. The L4 LoadBalancers (crafty-mc, unifi) publish an A record
# at their own LB VIP; Cloudflare's proxy is HTTP(S)-only, so a proxied A for a game or UniFi
# inform port is broken by construction. Each carries `cloudflare-proxied: "false"` and stays
# DNS-only, exactly as before this change. conformance/tests/l0-static/
# internal-services-publish-no-public-dns (assertion 9b) FAILS on any `service`-source publisher
# without it, so a new L4 Service cannot silently become a proxied record.
# #
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
# (b) the address is kept current IN-CLUSTER. Implementation:
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
# each published name becomes a CNAME to that one hostname;
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
# instance never owns or garbage-collects it.
# Internal services are unchanged: their routes carry `controller: none` (above), which is # Internal services are unchanged: their routes carry `controller: none` (above), which is
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4 # checked BEFORE the target is read, so they still publish nothing. Plex's web UI and Headlamp
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its # joined that set on 2026-11-08 (operator: LAN/VPN-only). `wan.${BASE_DOMAIN}` (ddns.yaml) has no
# LB VIP. Whether those should go public is a separate, still-open question. # external-dns TXT registry entry and is untouched; it now serves only WireGuard udp/51820.
#
# THE RISK THIS ADDS, AND WHAT GUARDS IT. If the annotation prefix ever stops being honoured
# again (the 2026-11-07 v0.23.0 incident), `target` is ignored and every name falls back to the
# Gateway's own address — and with this flag it would be published as a PROXIED A record at
# ${GATEWAY_VIP} (RFC1918; Cloudflare 1003/522s for every public name) while `controller: none`
# is also ignored and internal names publish. Assertion (8) of the L0 check above ties the
# prefix this tree writes to `values.annotationPrefix`; (8b) forbids overriding it here.
extraArgs:
- --cloudflare-proxied
# --------------------------------------------------------------------------------------------- # ---------------------------------------------------------------------------------------------
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below. # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
# #
+6 -2
View File
@@ -30,5 +30,9 @@ east-west default-deny (docs/09).
- **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09); - **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09);
add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS. add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS.
- **Public exposure** is a **manual edge port-forward** of 80/443 to the Gateway VIP (single SOHO - **Public exposure** is via the **Cloudflare Tunnel** (2026-11-08, `platform/cloudflared`): the
router, docs/09) — a documented manual step, the analogue of the old pfSense exception. Gateway's `external-dns.alpha.kubernetes.io/target` is `<tunnel-uuid>.cfargotunnel.com`, so every
published name is a proxied CNAME to the tunnel and cloudflared forwards to this Gateway's
ClusterIP. No inbound WAN 80/443 forward is needed (the old manual edge port-forward is retired —
runbooks/cloudflare-tunnel.md). LAN/VPN-only names (internal apps, plex, headlamp) carry
`controller: none` on their route and are 404'd by the tunnel.
+24 -12
View File
@@ -8,22 +8,34 @@ metadata:
name: shared name: shared
namespace: gateway namespace: gateway
annotations: annotations:
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every # PUBLIC DNS TARGET = THE CLOUDFLARE TUNNEL (2026-11-08, operator decision; replaces the
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is # 2026-10-03 `wan.${BASE_DOMAIN}` target). external-dns v0.23.0's gateway-httproute source reads
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex) # `target` from the GATEWAY only (source/gateway.go: `overrides: annotations.
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps # TargetsFromTargetAnnotation(gw.Annotations)`) — never from an HTTPRoute — so this one value moves
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}. # EVERY published route on this Gateway at once. Each published `<svc>.${BASE_DOMAIN}` (and the
# apex) becomes a CNAME to `<tunnel-uuid>.cfargotunnel.com`, and platform/externaldns/helmrelease.yaml
# sets `--cloudflare-proxied` so the CNAME is orange-clouded: public clients resolve Cloudflare
# anycast, Cloudflare terminates edge TLS (Total TLS) and hands the request to the in-cluster
# cloudflared connectors (platform/cloudflared), which forward to this Gateway's ClusterIP. No
# origin address of ours appears in public DNS, and no inbound WAN port is needed for HTTP(S).
# #
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the # THE UUID IS A LITERAL, AND IT MUST EQUAL `tunnel:` IN platform/cloudflared/config.yaml. A CNAME
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five # to any other <uuid>.cfargotunnel.com is a dead name (Cloudflare answers 1016). That equality is
# `exposure: public` apps. `internal` services stay unpublished via their # asserted on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks # (assertion 9a). A literal rather than a cluster-var keeps this change off clusters/**.
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never #
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml. # Scope: every route on this Gateway that publishes at all. `internal` services (and the
# operator-designated LAN/VPN-only names: plex, headlamp) stay unpublished via their
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
# #
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path. # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}" #
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
# it stays only as the WireGuard udp/51820 endpoint.
external-dns.alpha.kubernetes.io/target: "fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com"
spec: spec:
gatewayClassName: cilium gatewayClassName: cilium
listeners: listeners:
+7
View File
@@ -18,6 +18,13 @@ metadata:
annotations: annotations:
homelab.keithjackson.dev/auth-mode: oidc homelab.keithjackson.dev/auth-mode: oidc
homelab.keithjackson.dev/auth-groups: platform-admins homelab.keithjackson.dev/auth-groups: platform-admins
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
# resolving the name via PiHole's split-horizon wildcard (-> the Gateway VIP). The Cloudflare
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
external-dns.alpha.kubernetes.io/controller: none
spec: spec:
parentRefs: parentRefs:
- name: shared - name: shared
+36 -10
View File
@@ -31,10 +31,29 @@ The operator decisions it encodes (2026-11-08):
policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
`cloudflared-ingress-rules` fails the gate if that ever changes. `cloudflared-ingress-rules` fails the gate if that ever changes.
**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a **Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to
tunnel only when its record is a CNAME to `<UUID>.cfargotunnel.com`. Phase 1 (this component) `<UUID>.cfargotunnel.com`.** Phase 1 created no such record.
creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts
firing means "the tunnel is not ready", not "users are down". **Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes
every gateway name as a proxied CNAME to the tunnel: the shared Gateway's
`external-dns.alpha.kubernetes.io/target` is `<UUID>.cfargotunnel.com` and the HelmRelease sets
`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are
disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only
A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check
`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's
UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public
users are down** — every public HTTP(S) name depends on the connectors.
### After the Phase 2 merge — operator step (router)
Once `dig +short @1.1.1.1 <public name>` returns Cloudflare anycast addresses for the public names
(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:
1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP.
2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400**
(Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`).
3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and
`curl -m5 https://<WAN IP>/` now times out instead of answering.
Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
`CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization `CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
@@ -103,8 +122,8 @@ whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
- **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one. - **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
**external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`); **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
a record created behind its back either fights it or is deleted by it, and a real hostname a record created behind its back either fights it or is deleted by it, and a real hostname
routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2 routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns
change. The only record this runbook creates is the hand-made canary in step 6, and you delete it. (Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
- Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is - Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2. Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.
@@ -300,6 +319,9 @@ is deleted, which is why the last step is not optional.
``` ```
OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml) OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
# Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
# the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
``` ```
3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the 3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
@@ -315,8 +337,9 @@ is deleted, which is why the last step is not optional.
cf tunnel delete <old name> cf tunnel delete <old name>
cf tunnel list # the old tunnel is gone cf tunnel list # the old tunnel is gone
``` ```
After the Phase 2 DNS flip, records point at `<UUID>.cfargotunnel.com`: confirm external-dns Since Phase 2, records point at `<UUID>.cfargotunnel.com`: confirm external-dns has repointed
has repointed every record to the NEW UUID before deleting, or those hostnames go dark. every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those
hostnames go dark.
Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk. Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
## Rollback ## Rollback
@@ -332,6 +355,9 @@ is deleted, which is why the last step is not optional.
``` ```
Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
`flux resume kustomization cloudflared`. `flux resume kustomization cloudflared`.
- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records - **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
(that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it (Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
are **re-opened** first (they are closed after Phase 2). Never edit the records by hand:
external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked. in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
+17 -3
View File
@@ -11,6 +11,19 @@ env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway UR
**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
procedures with a manual last mile. procedures with a manual last mile.
## The Plex web UI is LAN/VPN-only (2026-11-08)
When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the
operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the
`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record
is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI
at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer.
**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands
clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the
WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after
the tunnel cut-over. Remote users use app.plex.tv, not our hostname.
## What was actually broken (measured 2026-10-03) ## What was actually broken (measured 2026-10-03)
Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
@@ -44,9 +57,10 @@ either way — this repo has no such vantage point and must not pretend otherwis
router by hand, so it must never move on a reschedule. router by hand, so it must never move on a reschedule.
While in the router, **check what else is bound to the WAN interface.** If remote management is While in the router, **check what else is bound to the WAN interface.** If remote management is
listening on 443 (see the certificate measurement above), move it or disable it — otherwise the listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this
Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since
though remote *streaming* works. These are two separate paths and they fail independently. 2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the
80/443 forwards are retired; only tcp/32400 matters here.)
Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous: Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous: