feat(dns): publish public names as proxied CNAMEs to the Cloudflare Tunnel

Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally,
L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and
Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns
gains assertion (9) tying the Gateway target to the tunnel UUID, requiring
cloudflare-proxied=false on every service-source publisher, and keeping the
tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture
opts out of proxying.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

runbooks/cloudflare-tunnel.md

@@ -31,10 +31,29 @@ The operator decisions it encodes (2026-11-08):
    policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
    `cloudflared-ingress-rules` fails the gate if that ever changes.
 
-**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a
-tunnel only when its record is a CNAME to `<UUID>.cfargotunnel.com`. Phase 1 (this component)
-creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts
-firing means "the tunnel is not ready", not "users are down".
+**Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to
+`<UUID>.cfargotunnel.com`.** Phase 1 created no such record.
+
+**Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes
+every gateway name as a proxied CNAME to the tunnel: the shared Gateway's
+`external-dns.alpha.kubernetes.io/target` is `<UUID>.cfargotunnel.com` and the HelmRelease sets
+`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are
+disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only
+A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check
+`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's
+UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public
+users are down** — every public HTTP(S) name depends on the connectors.
+
+### After the Phase 2 merge — operator step (router)
+
+Once `dig +short @1.1.1.1 <public name>` returns Cloudflare anycast addresses for the public names
+(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:
+
+1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP.
+2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400**
+   (Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`).
+3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and
+   `curl -m5 https://<WAN IP>/` now times out instead of answering.
 
 Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
 `CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
@@ -103,8 +122,8 @@ whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
 - **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
   **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
   a record created behind its back either fights it or is deleted by it, and a real hostname
-  routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2
-  change. The only record this runbook creates is the hand-made canary in step 6, and you delete it.
+  routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns
+  (Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
 - Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
   Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.
 
@@ -300,6 +319,9 @@ is deleted, which is why the last step is not optional.
    ```
    OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
    sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
+   # Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
+   # the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
+   sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
    ```
 3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
    same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
@@ -315,8 +337,9 @@ is deleted, which is why the last step is not optional.
    cf tunnel delete <old name>
    cf tunnel list                 # the old tunnel is gone
    ```
-   After the Phase 2 DNS flip, records point at `<UUID>.cfargotunnel.com`: confirm external-dns
-   has repointed every record to the NEW UUID before deleting, or those hostnames go dark.
+   Since Phase 2, records point at `<UUID>.cfargotunnel.com`: confirm external-dns has repointed
+   every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those
+   hostnames go dark.
    Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
 
 ## Rollback
@@ -332,6 +355,9 @@ is deleted, which is why the last step is not optional.
   ```
   Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
   `flux resume kustomization cloudflared`.
-- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records
-  (that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it
+- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
+  (Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
+  rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
+  are **re-opened** first (they are closed after Phase 2). Never edit the records by hand:
+  external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
   in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.

runbooks/plex-remote-access.md

@@ -11,6 +11,19 @@ env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway UR
 **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
 procedures with a manual last mile.
 
+## The Plex web UI is LAN/VPN-only (2026-11-08)
+
+When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the
+operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the
+`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record
+is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI
+at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer.
+
+**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands
+clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the
+WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after
+the tunnel cut-over. Remote users use app.plex.tv, not our hostname.
+
 ## What was actually broken (measured 2026-10-03)
 
 Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
@@ -44,9 +57,10 @@ either way — this repo has no such vantage point and must not pretend otherwis
 router by hand, so it must never move on a reschedule.
 
 While in the router, **check what else is bound to the WAN interface.** If remote management is
-listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
-Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
-though remote *streaming* works. These are two separate paths and they fail independently.
+listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this
+used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since
+2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the
+80/443 forwards are retired; only tcp/32400 matters here.)
 
 Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous: