Merge pull request 'feat(dns): public names via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only' (#156) from feat/dns-via-cloudflare-tunnel into main

Reviewed-on: #156
Reviewed-by: Keith Jackson <[email protected]>

apps/vms/unifi/50-service-l4.yaml

@@ -100,6 +100,11 @@ metadata:
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
+    # DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
+    # gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
+    # orange-clouded A for these L4 ports would be broken. Required by the L0 check
+    # internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
+    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Cluster

apps/wave-0/plex.yaml

@@ -302,6 +302,8 @@ metadata:
   namespace: plex
   labels:
     app.kubernetes.io/name: plex
+  annotations:
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

catalog/plex.yaml

@@ -347,7 +347,14 @@ lifecycle:
 # `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
 hardware:
   gpu: intel-quicksync
-exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
+# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
+# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
+# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
+# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
+# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
+# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
+# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
+exposure: internal            # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config

conformance/tests/l0-static/cloudflared-ingress-rules/chainsaw-test.yaml

@@ -39,7 +39,9 @@ spec:
         Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's
         target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the
         excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and
-        run the evaluator on mutated configs to prove it is not vacuous.
+        run the evaluator on mutated configs to prove it is not vacuous; and assert the
+        credentials Secret volume is readable by the container's run-as uid/groups (fsGroup +
+        mode), self-tested by mutation.
       try:
         - script:
             content: bash ./check.sh

conformance/tests/l0-static/cloudflared-ingress-rules/check.sh

@@ -26,9 +26,19 @@
 #       (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in
 #           platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its
 #           SOPS decryption patch — so a missing or invalid credential can stall nothing else;
-#   AND (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
+#       (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
 #           (an API rule added, plex moved below the wildcard, the catch-all removed) and must
-#           flag each one.
+#           flag each one;
+#   AND (7) every Secret volume the container mounts — the tunnel credentials above all — is
+#           READABLE by the container's run-as identity, modelled the way the kubelet lays it out:
+#           files owned by uid 0, group-owned by fsGroup if set (else gid 0), with the item mode or
+#           defaultMode (API default 0644). The identity is the container securityContext over
+#           the pod's; the process's groups are runAsGroup + fsGroup + supplementalGroups. An
+#           unset runAsUser is a FAIL, not a pass: the image default cannot be read offline. The
+#           model is deliberately conservative — it does not credit the kubelet's habit of
+#           OR-ing group-read into fsGroup-owned volumes, so 0400+fsGroup fails here though it
+#           would work. Self-tested by mutation (fsGroup removed; mode 0400; runAsUser unset).
+#           This is the 2026-11-08 CrashLoop: runAsUser 65532, no fsGroup, 0440 root:root.
 #
 # NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel,
 # that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are
@@ -236,6 +246,69 @@ check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.de
 check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False,
       "(4) the pods mount no ServiceAccount token")
 
+# (7) the credentials Secret is readable by the run-as identity
+def unreadable_secret_volumes(pod_spec):
+    """Return a description per Secret volume the containers mount but cannot read."""
+    bad = []
+    psc = pod_spec.get("securityContext") or {}
+    vols = {v["name"]: v for v in pod_spec.get("volumes") or [] if "secret" in v}
+    for c in (pod_spec.get("initContainers") or []) + (pod_spec.get("containers") or []):
+        csc = c.get("securityContext") or {}
+        uid = csc.get("runAsUser", psc.get("runAsUser"))
+        gid = csc.get("runAsGroup", psc.get("runAsGroup"))
+        fs_group = psc.get("fsGroup")
+        groups = {g for g in [gid, fs_group, *(psc.get("supplementalGroups") or [])]
+                  if g is not None}
+        for vm in c.get("volumeMounts") or []:
+            v = vols.get(vm["name"])
+            if v is None:
+                continue
+            sec = v["secret"]
+            if uid is None:
+                bad.append(f"container {c['name']} mounts Secret volume {vm['name']} but sets no "
+                           f"runAsUser (pod or container) — readability cannot be proven offline")
+                continue
+            default = sec.get("defaultMode", 0o644)
+            modes = [it.get("mode", default) for it in sec.get("items") or []] or [default]
+            owner_gid = fs_group if fs_group is not None else 0
+            for mode in modes:
+                readable = ((uid == 0 and mode & 0o400) or (owner_gid in groups and mode & 0o040)
+                            or mode & 0o004)
+                if not readable:
+                    bad.append(f"container {c['name']} (uid {uid}, groups {sorted(groups)}) cannot "
+                               f"read Secret volume {vm['name']} ({sec.get('secretName')}): files "
+                               f"are 0:{owner_gid} mode {mode:04o} — set pod fsGroup to a group "
+                               f"the container runs with, keeping 0440")
+    return bad
+
+
+pod_spec = dep["spec"]["template"]["spec"]
+creds_vol = [v for v in pod_spec.get("volumes") or []
+             if (v.get("secret") or {}).get("secretName") == "cloudflared-credentials"]
+check(len(creds_vol) == 1, "(7) the Deployment mounts Secret cloudflared-credentials as a volume")
+for v in unreadable_secret_volumes(pod_spec):
+    check(False, f"(7) {v}")
+sec_mutants = {}
+m = copy.deepcopy(pod_spec)
+m["securityContext"].pop("fsGroup", None)
+m["securityContext"].pop("supplementalGroups", None)
+sec_mutants["fsGroup removed"] = m
+m = copy.deepcopy(pod_spec)
+for v in m["volumes"]:
+    if "secret" in v:
+        v["secret"]["defaultMode"] = 0o400
+        for it in v["secret"].get("items") or []:
+            it["mode"] = 0o400
+sec_mutants["mode 0400"] = m
+m = copy.deepcopy(pod_spec)
+m["securityContext"].pop("runAsUser", None)
+for c in m["containers"]:
+    (c.get("securityContext") or {}).pop("runAsUser", None)
+sec_mutants["runAsUser unset"] = m
+for name, mut in sec_mutants.items():
+    check(unreadable_secret_volumes(mut),
+          f"(7) self-test: the readability model did NOT flag the mutant '{name}' — vacuous")
+
 if CHECKS < 25:
     FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
 if FAILURES:

conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml

@@ -92,9 +92,10 @@
 #     the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
 #   - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
 #     each publishes so the claim is reviewable, which is all a static check can offer.
-#   - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
-#     rather than a routable address; that is an open operator decision recorded in
-#     platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
+#   - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
+#     Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
+#     `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
+#     then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
 #   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
 # =============================================================================
 apiVersion: chainsaw.kyverno.io/v1alpha1

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -29,7 +29,17 @@
 #       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
 #   (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
 #       prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
-#       the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime.
+#       the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
+#   (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
+#       (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
+#           platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
+#       (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
+#           DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
+#       (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
+#           names it, and the only `service`-source publishers of one are the declared L4 DNS-only
+#           names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
+#           tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
+#       (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
 #
 # WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
 #   - That the record is gone from Cloudflare. Live read; belongs beside
@@ -39,8 +49,11 @@
 #     both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
 #   - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
 #     This check owns the NAME axis only.
-#   - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
-#     routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
+#   - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
+#     proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
+#     returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
+#     honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
+#     Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
 #   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
 #
 # Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
@@ -141,9 +154,12 @@ def docs_of(p: Path):
 # unadjudicated set cannot grow without a review event. Same ratchet reasoning as
 # KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
 #
-# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
-# the honest state after this change, and it is deliberate: converting them is an operator
-# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
+# WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
+# PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
+# address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
+# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
+# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
+# remaining name should be public at all stays an operator decision per name, not a renderer change,
 # and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
 # platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
 #
@@ -151,27 +167,28 @@ def docs_of(p: Path):
 # =============================================================================================
 KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
     "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
-        "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
+        "publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
         "platform component, not a catalog service; it must be reachable by every browser that "
         "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
         "whether the five public services are — an operator decision, not a renderer one.",
     "platform/keycloak/httproute.yaml:keycloak/keycloak":
-        "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
+        "publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
         "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
         "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
         "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
         "is meant to be public either way; it is listed because the PUBLISHING object is the "
         "hand-authored platform route, which carries no exposure field.",
     "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
-        "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
-        "certainly should not be publicly resolvable, but it is not a catalog service and "
-        "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
-        "internal services (Pi-hole must answer first).",
+        "publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
+        "behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
+        "name (it stayed public in the 2026-11-08 tunnel decision).",
     "platform/headlamp/httproute.yaml:headlamp/headlamp":
-        "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
-        "Keycloak OIDC; same adjudication as grafana.",
+        "publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
+        "decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
+        "with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
+        "disowned by (9d).",
     "platform/homepage/httproute.yaml:homepage/homepage":
-        "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
+        "publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
         "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
         "declaration.",
     "runbooks/serving/httproute.yaml:runbooks/runbooks":
@@ -180,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
         "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
         "header for why gating runbooks on Keycloak is a loop with no exit.",
     "platform/crafty/httproute.yaml:crafty/crafty":
-        "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
+        "publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
         "oauth2-proxy sidecar). Platform component, no exposure field.",
     "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
         "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
         "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
         "Gateway cannot express. A game server is the one case where a public name may be "
-        "genuinely wanted; the address it answers with is still RFC1918.",
+        "genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
     "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
         "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
-        "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
+        "(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
         "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
         "vm-demo, which declares no ingressRoutes and publishes nothing.",
 }
@@ -515,6 +532,113 @@ check(not bad_render,
       f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
       f"reads {ANNOTATION_PREFIX!r}")
 
+# =============================================================================================
+# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
+# =============================================================================================
+CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
+GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
+TARGET_KEY = ANNOTATION_PREFIX + "target"
+PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
+UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
+
+# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
+# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
+# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
+L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
+# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
+# deleting one from config.yaml cannot quietly also release it from (9c).
+OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
+
+cf = next(iter(docs_of(CF_CONFIG)), None)
+check(cf is not None and isinstance(cf.get("ingress"), list),
+      f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
+      f"the tunnel refuses or which tunnel the Gateway must target")
+cf = cf or {}
+tunnel = str(cf.get("tunnel") or "")
+check(bool(UUID_RE.match(tunnel)),
+      f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
+      f"the Gateway target is derived from it")
+gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
+           and ident(d) == ("gateway", "shared")), None)
+check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
+want_target = f"{tunnel}.cfargotunnel.com"
+got_target = annotations_of(gw or {}).get(TARGET_KEY)
+check(got_target == want_target,
+      f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
+      f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
+      f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
+      f"an origin address of ours.")
+check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
+      or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
+      "(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
+      "CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
+
+for rel, d in dns_services:
+    if is_disowned(d):
+        continue
+    if HOSTNAME_KEY not in annotations_of(d):
+        continue
+    v = annotations_of(d).get(PROXIED_KEY)
+    check(str(v).lower() == "false",
+          f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
+          f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
+          f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
+          f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
+for rel, d in routes:
+    v = annotations_of(d).get(PROXIED_KEY)
+    check(v is None or str(v).lower() == "true",
+          f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
+          f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
+          f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
+
+refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
+           if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
+check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
+                        "operator's LAN/VPN-only list is known to be non-empty)")
+published_l4: set[str] = set()
+for rel, d in routes:
+    hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
+    leaked = sorted(hosts & refused)
+    if is_disowned(d):
+        unrefused = sorted(hosts - refused)
+        check(not unrefused,
+              f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
+              f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
+              f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
+        continue
+    check(not leaked,
+          f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
+          f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
+          f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
+for rel, d in dns_services:
+    if is_disowned(d):
+        continue
+    hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
+             if h.strip()}
+    for h in sorted(hosts & refused):
+        published_l4.add(h)
+        check(h in L4_DNS_ONLY_NAMES,
+              f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
+              f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
+check(L4_DNS_ONLY_NAMES <= refused,
+      f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
+      f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
+check(published_l4 == L4_DNS_ONLY_NAMES,
+      f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
+      f"published by any `service`-source publisher — delete them from the list")
+
+check(OPERATOR_LAN_ONLY_NAMES <= refused,
+      f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
+      f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
+for name in sorted(OPERATOR_LAN_ONLY_NAMES):
+    naming = [(rel, d) for rel, d in routes
+              if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
+    check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
+    for rel, d in naming:
+        check(is_disowned(d),
+              f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
+              f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
+
 # =============================================================================================
 if FAILURES:
     for f in FAILURES:

conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml

@@ -64,6 +64,12 @@ spec:
                 annotations:
                   # conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
                   external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
+                  # DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
+                  # gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
+                  # record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
+                  # and the exact-IP poll below can never match. Same opt-out every real
+                  # `service`-source publisher carries (unifi, crafty-mc).
+                  external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
               spec:
                 type: LoadBalancer
                 # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and

decisions/2026-11-08-cloudflared-creds-permission.md

@@ -0,0 +1,59 @@
+---
+date: 2026-11-08
+branch: fix/cloudflared-creds-permission
+title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
+---
+
+## What changed
+
+- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
+  the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
+  Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
+  `root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
+  `couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
+  and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
+  `fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
+  mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
+  Still non-root, `runAsNonRoot: true`, unchanged otherwise.
+- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
+  Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
+  mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
+  (runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
+  runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
+  kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
+  runAsUser unset must each be flagged. Chainsaw step description updated to match.
+- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
+  credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
+  note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
+  swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
+  has none), then `cloudflared tunnel cleanup`/`delete <old>`.
+- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
+  operator is rotating the credential (it was exposed in a transcript) in a separate PR.
+
+## What I verified
+
+- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
+  `{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
+  `fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
+  restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
+- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
+  `User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
+- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
+  `container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
+  0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
+- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
+  fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
+- Toolbox gate: see the PR description for the rc and skipped stages.
+
+## What I did NOT verify
+
+- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
+  file mode inside the running container was not observed (pods crash before an exec is possible,
+  and exec would be an operator action anyway). The readability claim rests on the kubelet's
+  documented fsGroup behaviour and the identical, working keycloak reconciler.
+- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
+  the operator is rotating it.
+- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
+  syntax is from the CLI's documented commands, not run here.
+- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
+  image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.

decisions/2026-11-08-dns-via-cloudflare-tunnel.md

@@ -0,0 +1,77 @@
+---
+date: 2026-11-08
+branch: feat/dns-via-cloudflare-tunnel
+title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard
+---
+
+## What changed
+
+One atomic change, so the conformance guard lands with the DNS flip.
+
+- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target`
+  is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was
+  `wan.${BASE_DOMAIN}`). Comment block rewritten.
+- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because
+  v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The
+  public-target comment block rewritten, including the prefix-regression risk.
+- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`:
+  `external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They
+  are the only `service`-source publishers in the tree and live (measured).
+- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml`
+  re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The
+  render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none`
+  on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files
+  were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched.
+- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`.
+- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and
+  register prose updated; new assertion (9):
+  (9a) Gateway target == `<tunnel: from platform/cloudflared/config.yaml>.cfargotunnel.com`, and
+  `--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied:
+  "false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the
+  tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set
+  {unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept
+  consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose
+  updated.
+- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets
+  `cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails.
+- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`,
+  `platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md`
+  (Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation
+  now also swaps the Gateway UUID; rollback path).
+
+## What I verified
+
+- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from
+  `gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations;
+  `annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the
+  one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a
+  parsed `"false"` annotation overrides it.
+- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation
+  confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails.
+- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target
+  back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex
+  golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c).
+  `cloudflared-ingress-rules` still passes; `render.py --selftest` PASS.
+- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry
+  `controller: none`; plex and headlamp do not (yet); the only Services with an external-dns
+  hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently
+  `wan.home.keithjackson.dev`.
+- Toolbox gate result: see the PR description (rc and skipped stages quoted there).
+
+## What I did NOT verify
+
+- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs
+  to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched.
+  Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`.
+- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs`
+  is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it.
+- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through
+  Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate,
+  but Cloudflare bot/WAF settings on that path are untested.
+- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class),
+  `target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal
+  names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it
+  cannot catch a controller that ignores the flag. No runtime alert exists for it.
+- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR.
+- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges;
+  `platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME.

platform/cloudflared/cloudflared-credentials.sops.yaml

@@ -9,7 +9,7 @@ metadata:
     namespace: cloudflared
 type: Opaque
 stringData:
-    credentials.json: ENC[AES256_GCM,data:PM5pgIW/JEUMLuhApkPhgMOkJCL0nrlgLBb3tG+pHdm8JWdNK2yYCZUWM3USY7Uy2IYphZh35RwPOrWSUCyBlSq+TeEKBkzTTgJhp4gqaQ==,iv:Sr0Z6d9LmHvGN5oebIf+YBbFyLIdNOi80d6WwOF3PtU=,tag:Enb5/9v7H1nu2kvh4zEjXg==,type:str]
+    credentials.json: ENC[AES256_GCM,data:68G438/jsM4KqPt/3wjFp6jVmAVnsidx/FEdK6Lke2euQgkDqRcJjhoKZurapgP8zsP+FrQ98VgsQgUpERHXUCRuJp5wRIgTVmL23CweuZLacaLxBo6eoKmNC/H5DzGZsyhDxGjcXc+8BGpfDFzHR39Uo2h56XstiHtyZhnhlhia8nprLip9RinA9mbKRMJKTR/6IsZsdc91E22aEycDjZ3+nYmr4KP2+km3fd15Nw==,iv:CuAUscGmvMqT2ixFJZVzJWSvDYwbfQl79sQ+FHnXdVQ=,tag:H4fFjXPCFoxE1Mqwx0R5sQ==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -25,8 +25,8 @@ sops:
             emhHaFpEZWpMN0JXL2NVaW5PU1ZwVW8KkCDX7ED039X12BCmiSqHr0fmZKSZmfjP
             DIiR6DvWWw1TcB3CjD5q29nIthCDf9zlrpuMfQ+9ODAlRr5t6zP+cg==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-11-08T18:28:24Z"
-    mac: ENC[AES256_GCM,data:RGiftdP4Bd5Qwyfx/pXEyZTBzTvr5Tvc7k0foEzTQGiSg6p0meurwtmCe/gMjKUZLhwf/0xqi1DtPQGS6UY8m8YFq1v1nXtohUQWLPXgTukMT915tHmohZeG91wYdyG9kA5E03udn+QNdOiI8OtjKoqAtifBoymCYd/EDOzWLQU=,iv:0vDIoDJ4QGECq7YjUPIlsU1j/AHfr1W32/+RjEOfpWo=,tag:BWT7nxiUxkVhEDh0JiE1Eg==,type:str]
+    lastmodified: "2026-11-08T22:05:00Z"
+    mac: ENC[AES256_GCM,data:FaTWQhcdYD4sDu50K9NisBO81cOwHqWKt9b/0XBvZzV5FAMNHBIi4Md3by5XT16paMuS+kgz20/+CLX1PkJ6FNbCwnp7M0+a+xtT+UT67apgJ04B6GW7HZJ+iUS6sOIIFoefSDHfKpO4bYLDIoXWupLYFlbHHRJQqQXpU7yvVgI=,iv:YDSYmgc7YUsYVeh7tPm8Sztvr3+L6xb1lww4lZdzDaY=,tag:7IektWvjJpKG98edUqNadQ==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

platform/cloudflared/config.yaml

@@ -20,7 +20,7 @@
 # OPERATOR STEP: replace REPLACE-ME-TUNNEL-UUID with the UUID printed by `cloudflared tunnel create`
 # (runbooks/cloudflare-tunnel.md step 3). Until then cloudflared treats the value as a tunnel NAME,
 # finds no origin certificate to resolve it with, and exits — a CrashLoop in a non-gating tier.
-tunnel: REPLACE-ME-TUNNEL-UUID
+tunnel: fc81938b-5855-47f1-8a27-6286f56dda93
 credentials-file: /etc/cloudflared/creds/credentials.json
 # Serves /metrics (Prometheus), /ready (200 only with >=1 live edge connection) and /healthcheck.
 metrics: 0.0.0.0:2000

platform/cloudflared/deployment.yaml

@@ -47,6 +47,14 @@ spec:
         # The image is distroless `nonroot` (65532).
         runAsUser: 65532
         runAsGroup: 65532
+        # fsGroup makes the kubelet GROUP-own the Secret volume's files to 65532. Without it they
+        # are root:root, and defaultMode 0440 (below) is unreadable to uid 65532 — every replica
+        # CrashLooped on "couldn't read tunnel credentials ... permission denied" (2026-11-08),
+        # which cloudflared then misreports as "Error locating origin cert". Same lesson as
+        # platform/keycloak/client-secret-reconciler.yaml. fsGroup + 0440 keeps the credential
+        # unreadable to any other uid rather than relaxing it to world-readable 0444.
+        # Pinned by the L0 check cloudflared-ingress-rules, assertion (7).
+        fsGroup: 65532
         seccompProfile:
           type: RuntimeDefault
       topologySpreadConstraints:
@@ -121,6 +129,8 @@ spec:
           secret:
             # cloudflared-credentials.sops.yaml — the tunnel credentials JSON, by name only.
             secretName: cloudflared-credentials
+            # 0440, owner root, group fsGroup 65532 (pod securityContext) — readable by the
+            # container user through its group, by nobody else.
             defaultMode: 0440
 ---
 # One replica may be disrupted at a time — enough for kured's one-node-at-a-time drain to

platform/crafty/service-l4.yaml

@@ -101,6 +101,11 @@ metadata:
     lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
     # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
     external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
+    # DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
+    # gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
+    # orange-clouded A for these L4 ports would be broken. Required by the L0 check
+    # internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
+    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Cluster

platform/externaldns/README.md

@@ -20,29 +20,44 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
 its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
 Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
 
-## Public target + in-cluster DDNS (2026-10-03)
+## Public target: the Cloudflare Tunnel, proxied (2026-11-08)
 
-Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
+Gateway-published records are **proxied CNAMEs to the Cloudflare Tunnel**
+(`platform/cloudflared`), not the WAN IP and not the Gateway VIP:
 
 ```
-plex.home.keithjackson.dev  CNAME  wan.home.keithjackson.dev   <- external-dns (Gateway `target` annotation)
-wan.home.keithjackson.dev   A      <current WAN IP>            <- cloudflare-ddns (ddns.yaml), every 5m
+gitea.home.keithjackson.dev  CNAME (proxied)  fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com  <- external-dns
+unifi.home.keithjackson.dev  A     (DNS-only) 192.168.2.21    <- external-dns, `service` source (L4)
+crafty-mc.home...            A     (DNS-only) 192.168.2.22    <- external-dns, `service` source (L4)
+wan.home.keithjackson.dev    A     (DNS-only) <current WAN IP> <- cloudflare-ddns (ddns.yaml): WireGuard udp/51820 ONLY
 ```
 
-- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
-  applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
-  nothing.
-- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
-  It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
-  `egress-class: vpn`.
+- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`); external-dns
+  reads it from the Gateway only, so it applies to every route that publishes at all. Its UUID must
+  equal `tunnel:` in `platform/cloudflared/config.yaml` (L0 check
+  `internal-services-publish-no-public-dns` (9a)).
+- `--cloudflare-proxied` (HelmRelease `extraArgs`) orange-clouds every record by default — the only
+  practical lever, because v0.23.0 reads the per-object `cloudflare-proxied` annotation from the
+  route/Service, never the Gateway. **Every `service`-source publisher must carry
+  `external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"`** (L4 ports cannot ride the HTTP
+  proxy); (9b) fails the gate otherwise.
+- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
+  set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
+- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
 - LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
   (split-horizon) and never use the public path.
-- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
-  OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
-  the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
+- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
+  OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
+  fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
+  router NAT hairpin is no longer on that path.
 
-Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
-`kubectl -n external-dns logs deploy/cloudflare-ddns`.
+Check: `dig +short @1.1.1.1 gitea.home.keithjackson.dev` → Cloudflare anycast addresses (no CNAME is
+visible for a proxied record); `kubectl -n external-dns logs deploy/external-dns`.
+
+### History
+
+2026-10-03 → 2026-11-08: records were DNS-only CNAMEs to `wan.home.keithjackson.dev`, with the router
+forwarding WAN tcp/80+443 to the Gateway VIP. Before 2026-10-03 they answered the Gateway VIP itself.
 
 ## Secrets
 

platform/externaldns/ddns.yaml

@@ -2,9 +2,15 @@
 # In-cluster dynamic DNS for the home WAN (2026-10-03).
 #
 # Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
-# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
-# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
-# moves all of them through this one record, with no Git change and no external-dns involvement.
+# current public IP.
+#
+# WIREGUARD-ONLY SINCE 2026-11-08. From 2026-10-03 every public name external-dns published from the
+# shared Gateway was a CNAME to this record. They are now proxied CNAMEs to the Cloudflare Tunnel
+# (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so NOTHING in the zone
+# points here any more. The record stays for exactly one consumer: the WireGuard VPN endpoint
+# (udp/51820, platform/vpn-ingress, infra/vpn-ingress/new-peer.sh), which cannot ride Cloudflare's
+# HTTP proxy and needs the real WAN address. It must stay DNS-only (`PROXIED` unset/false below) —
+# an orange-clouded A record would hand peers a Cloudflare anycast address that drops UDP.
 #
 # OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
 # external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
@@ -64,7 +70,7 @@ spec:
             - name: IP6_PROVIDER
               value: "none"         # IPv4 WAN only; leaves any AAAA alone
             - name: PROXIED
-              value: "false"        # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
+              value: "false"        # DNS-only: WireGuard udp/51820 needs the real WAN IP
             - name: UPDATE_CRON
               value: "@every 5m"
             - name: DELETE_ON_STOP

platform/externaldns/helmrelease.yaml

@@ -107,29 +107,47 @@ spec:
     # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
     # this instance's view, so it cannot make prod capable of deleting a record it does not own.
     # ---------------------------------------------------------------------------------------------
-    # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
+    # THE PUBLIC TARGET — THE CLOUDFLARE TUNNEL, PROXIED (2026-11-08; supersedes 2026-10-03)
     #
-    # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
-    # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
-    # fix, and both still hold:
-    #   * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
-    #     from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
-    #     gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
-    #     shared Gateway therefore moves EVERY gateway-sourced record at once.
-    #   * The home WAN IP is residential and can change, so a literal IP in Git would rot.
+    # History: until 2026-10-03 every gateway-sourced record answered the Gateway VIP (RFC1918);
+    # from 2026-10-03 they were CNAMEs to `wan.${BASE_DOMAIN}` (DNS-only, the home WAN IP, with the
+    # router forwarding tcp/80+443 to the VIP). Both published an origin address of ours. Now:
+    #   * platform/gateway/gateway.yaml sets `target: <tunnel-uuid>.cfargotunnel.com` on the shared
+    #     Gateway. `target` is read in v0.23.0's gateway-httproute source ONLY from the GATEWAY
+    #     (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(gw.Annotations)`),
+    #     never from an HTTPRoute, so this one value moves every gateway-sourced record at once.
+    #   * `--cloudflare-proxied` below orange-clouds every record by default. It is GLOBAL because
+    #     that is the only practical lever: v0.23.0 reads the per-object `cloudflare-proxied`
+    #     annotation from the ROUTE (or Service), never the Gateway (source/gateway.go:
+    #     `annotations.ProviderSpecificAnnotations(annots)` with `annots := meta.Annotations` of the
+    #     route), so a Gateway-level opt-in does not exist. A CNAME to cfargotunnel.com MUST be
+    #     proxied — DNS-only it resolves to nothing routable — so "proxied by default" is the right
+    #     default for this zone's gateway names.
+    #   * The provider honours a per-object `"false"` over the global default (provider/cloudflare
+    #     shouldBeProxied: ParseBool of the annotation overrides proxiedByDefault). The key it reads
+    #     is `<annotationPrefix>cloudflare-proxied` (annotations.SetAnnotationPrefix re-derives
+    #     CloudflarePrefix), i.e. `external-dns.alpha.kubernetes.io/cloudflare-proxied` here.
+    #
+    # THE `service` SOURCE MUST OPT OUT. The L4 LoadBalancers (crafty-mc, unifi) publish an A record
+    # at their own LB VIP; Cloudflare's proxy is HTTP(S)-only, so a proxied A for a game or UniFi
+    # inform port is broken by construction. Each carries `cloudflare-proxied: "false"` and stays
+    # DNS-only, exactly as before this change. conformance/tests/l0-static/
+    # internal-services-publish-no-public-dns (assertion 9b) FAILS on any `service`-source publisher
+    # without it, so a new L4 Service cannot silently become a proxied record.
     #
-    # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
-    # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
-    # (b) the address is kept current IN-CLUSTER. Implementation:
-    #   * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
-    #     each published name becomes a CNAME to that one hostname;
-    #   * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
-    #     record at the current WAN IP. That record has no external-dns TXT registry entry, so this
-    #     instance never owns or garbage-collects it.
     # Internal services are unchanged: their routes carry `controller: none` (above), which is
-    # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
-    # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
-    # LB VIP. Whether those should go public is a separate, still-open question.
+    # checked BEFORE the target is read, so they still publish nothing. Plex's web UI and Headlamp
+    # joined that set on 2026-11-08 (operator: LAN/VPN-only). `wan.${BASE_DOMAIN}` (ddns.yaml) has no
+    # external-dns TXT registry entry and is untouched; it now serves only WireGuard udp/51820.
+    #
+    # THE RISK THIS ADDS, AND WHAT GUARDS IT. If the annotation prefix ever stops being honoured
+    # again (the 2026-11-07 v0.23.0 incident), `target` is ignored and every name falls back to the
+    # Gateway's own address — and with this flag it would be published as a PROXIED A record at
+    # ${GATEWAY_VIP} (RFC1918; Cloudflare 1003/522s for every public name) while `controller: none`
+    # is also ignored and internal names publish. Assertion (8) of the L0 check above ties the
+    # prefix this tree writes to `values.annotationPrefix`; (8b) forbids overriding it here.
+    extraArgs:
+      - --cloudflare-proxied
     # ---------------------------------------------------------------------------------------------
     # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
     #

platform/gateway/README.md

@@ -30,5 +30,9 @@ east-west default-deny (docs/09).
 
 - **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09);
   add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS.
-- **Public exposure** is a **manual edge port-forward** of 80/443 to the Gateway VIP (single SOHO
-  router, docs/09) — a documented manual step, the analogue of the old pfSense exception.
+- **Public exposure** is via the **Cloudflare Tunnel** (2026-11-08, `platform/cloudflared`): the
+  Gateway's `external-dns.alpha.kubernetes.io/target` is `<tunnel-uuid>.cfargotunnel.com`, so every
+  published name is a proxied CNAME to the tunnel and cloudflared forwards to this Gateway's
+  ClusterIP. No inbound WAN 80/443 forward is needed (the old manual edge port-forward is retired —
+  runbooks/cloudflare-tunnel.md). LAN/VPN-only names (internal apps, plex, headlamp) carry
+  `controller: none` on their route and are 404'd by the tunnel.

platform/gateway/gateway.yaml

@@ -8,22 +8,34 @@ metadata:
   name: shared
   namespace: gateway
   annotations:
-    # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
-    # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
-    # unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
-    # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
-    # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
+    # PUBLIC DNS TARGET = THE CLOUDFLARE TUNNEL (2026-11-08, operator decision; replaces the
+    # 2026-10-03 `wan.${BASE_DOMAIN}` target). external-dns v0.23.0's gateway-httproute source reads
+    # `target` from the GATEWAY only (source/gateway.go: `overrides: annotations.
+    # TargetsFromTargetAnnotation(gw.Annotations)`) — never from an HTTPRoute — so this one value moves
+    # EVERY published route on this Gateway at once. Each published `<svc>.${BASE_DOMAIN}` (and the
+    # apex) becomes a CNAME to `<tunnel-uuid>.cfargotunnel.com`, and platform/externaldns/helmrelease.yaml
+    # sets `--cloudflare-proxied` so the CNAME is orange-clouded: public clients resolve Cloudflare
+    # anycast, Cloudflare terminates edge TLS (Total TLS) and hands the request to the in-cluster
+    # cloudflared connectors (platform/cloudflared), which forward to this Gateway's ClusterIP. No
+    # origin address of ours appears in public DNS, and no inbound WAN port is needed for HTTP(S).
     #
-    # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
-    # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
-    # `exposure: public` apps. `internal` services stay unpublished via their
-    # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
-    # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
-    # from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
+    # THE UUID IS A LITERAL, AND IT MUST EQUAL `tunnel:` IN platform/cloudflared/config.yaml. A CNAME
+    # to any other <uuid>.cfargotunnel.com is a dead name (Cloudflare answers 1016). That equality is
+    # asserted on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns
+    # (assertion 9a). A literal rather than a cluster-var keeps this change off clusters/**.
+    #
+    # Scope: every route on this Gateway that publishes at all. `internal` services (and the
+    # operator-designated LAN/VPN-only names: plex, headlamp) stay unpublished via their
+    # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns
+    # applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
+    # ingress rules, so even a hand-made record cannot reach them through the tunnel.
     #
     # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
     # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
-    external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
+    #
+    # `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
+    # it stays only as the WireGuard udp/51820 endpoint.
+    external-dns.alpha.kubernetes.io/target: "fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com"
 spec:
   gatewayClassName: cilium
   listeners:

platform/headlamp/httproute.yaml

@@ -18,6 +18,13 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: oidc
     homelab.keithjackson.dev/auth-groups: platform-admins
+    # LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
+    # headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
+    # resolving the name via PiHole's split-horizon wildcard (-> the Gateway VIP). The Cloudflare
+    # Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
+    # Same mechanism the renderer emits for `exposure: internal` services; asserted by
+    # conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
     - name: shared

runbooks/cloudflare-tunnel.md

@@ -31,10 +31,29 @@ The operator decisions it encodes (2026-11-08):
    policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
    `cloudflared-ingress-rules` fails the gate if that ever changes.
 
-**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a
-tunnel only when its record is a CNAME to `<UUID>.cfargotunnel.com`. Phase 1 (this component)
-creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts
-firing means "the tunnel is not ready", not "users are down".
+**Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to
+`<UUID>.cfargotunnel.com`.** Phase 1 created no such record.
+
+**Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes
+every gateway name as a proxied CNAME to the tunnel: the shared Gateway's
+`external-dns.alpha.kubernetes.io/target` is `<UUID>.cfargotunnel.com` and the HelmRelease sets
+`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are
+disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only
+A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check
+`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's
+UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public
+users are down** — every public HTTP(S) name depends on the connectors.
+
+### After the Phase 2 merge — operator step (router)
+
+Once `dig +short @1.1.1.1 <public name>` returns Cloudflare anycast addresses for the public names
+(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:
+
+1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP.
+2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400**
+   (Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`).
+3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and
+   `curl -m5 https://<WAN IP>/` now times out instead of answering.
 
 Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
 `CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
@@ -63,23 +82,36 @@ patch is missing; see causes below.
 
 ## Likely causes, most likely first
 
-1. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
-   error about the tunnel `REPLACE-ME-TUNNEL-UUID` or a missing origin certificate. Do steps 1-5.
-2. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
+**Read the FIRST error line, not the last.** When cloudflared cannot use the credentials file it
+falls back to looking for an origin certificate, and the error it ends on —
+`Cannot determine default origin certificate path` / `error parsing tunnel ID: Error locating
+origin cert` — is a **red herring** whatever the real cause was. The credentials file is the
+whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
+
+1. **`permission denied` on `credentials.json` → the "origin cert" error is a red herring.** Logs:
+   `couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission
+   denied`, then the origin-cert lines. The container (uid 65532) cannot read the Secret file:
+   the pod `securityContext.fsGroup: 65532` is missing, so the 0440 file is `root:root`. Fix it in
+   git (`platform/cloudflared/deployment.yaml`); the L0 check `cloudflared-ingress-rules`,
+   assertion (7), should already have refused that. Do NOT loosen `defaultMode` to 0444. (This is
+   what CrashLooped every replica on 2026-11-08.)
+2. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
+   error about the tunnel `REPLACE-ME-TUNNEL-UUID` (and the origin-cert red herring). Do steps 1-5.
+3. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
    credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
-3. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
+4. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
    from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed
    it.
-4. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
+5. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
    ```
    infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
         hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
    ```
    (`ds/cilium` picks one node's agent; repeat per node if empty.)
-5. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
+6. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
    (step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
-6. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
+7. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
    node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.
 
 ## When to escalate or stop
@@ -90,8 +122,8 @@ patch is missing; see causes below.
 - **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
   **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
   a record created behind its back either fights it or is deleted by it, and a real hostname
-  routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2
-  change. The only record this runbook creates is the hand-made canary in step 6, and you delete it.
+  routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns
+  (Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
 - Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
   Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.
 
@@ -270,6 +302,46 @@ disappears with the pod on the next rollout.)
 Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone:
 `dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing.
 
+## Rotating the tunnel credential
+
+Do this whenever the credentials JSON (`TunnelSecret`) or `cert.pem` may have been exposed — a
+transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this
+workflow, so rotation is **a new tunnel**: the old credential stays valid until the old tunnel
+is deleted, which is why the last step is not optional.
+
+1. **Create the new tunnel** (*Enrolling*, step 2's `cf` helper; `cf tunnel login` again if
+   `cert.pem` is not to hand). Use a NEW name — the old one is still taken:
+   ```
+   cf tunnel list                       # note the OLD tunnel's id and name
+   cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2
+   ```
+2. **Swap the UUID** in your own worktree, on a new branch (CLAUDE.md §0):
+   ```
+   OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
+   sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
+   # Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
+   # the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
+   sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
+   ```
+3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
+   same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
+   `sops -d` to the terminal.
+4. **Merge** — gate, PR, `/approve-override` by the operator (it touches `**/*.sops.yaml`), merge.
+   Flux rolls the Deployment because the content-hashed ConfigMap changed.
+5. **Verify** with *Enrolling* step 6: both replicas Ready, `Registered tunnel connection` in
+   each replica's log, `cf tunnel info home-k3s-2` lists two connectors, and
+   `cf tunnel info <old name>` lists **none**.
+6. **Only then delete the old tunnel**, which revokes the exposed credential:
+   ```
+   cf tunnel cleanup <old name>   # drops any stale connector records first
+   cf tunnel delete <old name>
+   cf tunnel list                 # the old tunnel is gone
+   ```
+   Since Phase 2, records point at `<UUID>.cfargotunnel.com`: confirm external-dns has repointed
+   every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those
+   hostnames go dark.
+   Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
+
 ## Rollback
 
 - **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and
@@ -283,6 +355,9 @@ Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is
   ```
   Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
   `flux resume kustomization cloudflared`.
-- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records
-  (that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it
+- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
+  (Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
+  rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
+  are **re-opened** first (they are closed after Phase 2). Never edit the records by hand:
+  external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
   in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.

runbooks/plex-remote-access.md

@@ -11,6 +11,19 @@ env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway UR
 **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
 procedures with a manual last mile.
 
+## The Plex web UI is LAN/VPN-only (2026-11-08)
+
+When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the
+operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the
+`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record
+is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI
+at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer.
+
+**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands
+clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the
+WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after
+the tunnel cut-over. Remote users use app.plex.tv, not our hostname.
+
 ## What was actually broken (measured 2026-10-03)
 
 Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
@@ -44,9 +57,10 @@ either way — this repo has no such vantage point and must not pretend otherwis
 router by hand, so it must never move on a reschedule.
 
 While in the router, **check what else is bound to the WAN interface.** If remote management is
-listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
-Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
-though remote *streaming* works. These are two separate paths and they fail independently.
+listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this
+used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since
+2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the
+80/443 forwards are retired; only tcp/32400 matters here.)
 
 Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous: