# cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts Alerts that link here: **`CloudflaredTunnelDown`** (no replica has a live edge connection, or none is being scraped) and **`CloudflaredReplicasDegraded`** (fewer than 2 replicas Available). Component: `platform/cloudflared/` (Flux Kustomization `cloudflared`, `clusters/base/cloudflared.yaml`). Design record: `decisions/2026-11-08-cloudflare-tunnel.md`. Every step below is an **operator** step. An agent must not mint the tunnel credential, run `flux reconcile`, or mutate the cluster (CLAUDE.md §5, §6). ## What this means here cloudflared is an **outbound-only** connector: two replicas dial the Cloudflare edge on 7844 (QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway (`https://cilium-gateway-shared.gateway.svc.cluster.local:443`, SNI = the request's Host, so the Gateway's existing `wildcard-home-tls` Let's Encrypt certificate is what cloudflared validates). The tunnel is **locally managed**: its ingress rules are `platform/cloudflared/config.yaml`, in git, not in the Cloudflare dashboard. The operator decisions it encodes (2026-11-08): 1. Public hostnames stay `*.home.keithjackson.dev` and the apex; edge certificates come from Cloudflare **ACM + Total TLS**. 2. **Plex is not on the tunnel** (`plex`, `plex-vip` → 404). 3. **jackett, pihole, semaphore, qbittorrent, radarr, sonarr** and **headlamp** stay LAN/VPN-only, as do the L4 VIPs **unifi** and **crafty-mc** (all → 404 in cloudflared, and the LAN-only Gateway backends are also refused by the egress policy). Headlamp is an operator decision of 2026-11-08; the other six are the catalog's `exposure: internal` services. 4. **The Kubernetes API is never reachable through the tunnel.** No rule targets it and the egress policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check `cloudflared-ingress-rules` fails the gate if that ever changes. **Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to `.cfargotunnel.com`.** Phase 1 created no such record. **Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes every gateway name as a proxied CNAME to the tunnel: the shared Gateway's `external-dns.alpha.kubernetes.io/target` is `.cfargotunnel.com` and the HelmRelease sets `--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check `internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public users are down** — every public HTTP(S) name depends on the connectors. ### After the Phase 2 merge — operator step (router) Once `dig +short @1.1.1.1 ` returns Cloudflare anycast addresses for the public names (and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more: 1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP. 2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400** (Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`). 3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and `curl -m5 https:///` now times out instead of answering. Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and `CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization is `wait: false` and nothing depends on it. ## First commands ``` # Are the pods running, and on different nodes? infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide' # What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica. infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50' # Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)? infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n flux-system get kustomization cloudflared; kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \ -o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"' ``` Never print the Secret's values. `ENC[` in the decrypted Secret means the overlay's decryption patch is missing; see causes below. ## Likely causes, most likely first **Read the FIRST error line, not the last.** When cloudflared cannot use the credentials file it falls back to looking for an origin certificate, and the error it ends on — `Cannot determine default origin certificate path` / `error parsing tunnel ID: Error locating origin cert` — is a **red herring** whatever the real cause was. The credentials file is the whole of what this Deployment needs; no `cert.pem` ever belongs in the pod. 1. **`permission denied` on `credentials.json` → the "origin cert" error is a red herring.** Logs: `couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission denied`, then the origin-cert lines. The container (uid 65532) cannot read the Secret file: the pod `securityContext.fsGroup: 65532` is missing, so the 0440 file is `root:root`. Fix it in git (`platform/cloudflared/deployment.yaml`); the L0 check `cloudflared-ingress-rules`, assertion (7), should already have refused that. Do NOT loosen `defaultMode` to 0444. (This is what CrashLooped every replica on 2026-11-08.) 2. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an error about the tunnel `REPLACE-ME-TUNNEL-UUID` (and the origin-cert red herring). Do steps 1-5. 3. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command. 4. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed it. 5. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact' ``` (`ds/cilium` picks one node's agent; repeat per node if empty.) 6. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list` (step 2's podman form) no longer shows it. Re-enrol with steps 2-5. 7. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per node — a cordoned node during a kured reboot is normal and clears) or an image pull failed. ## When to escalate or stop - **Stop** if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently overridden by `config.yaml`. Use the emergency stop under *Rollback*, then fix in git. - **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one. **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`); a record created behind its back either fights it or is deleted by it, and a real hostname routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns (Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it. - Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2. --- ## Enrolling the tunnel (once) cloudflared is **not** in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image the Deployment uses, under podman — no host install. The working directory is `~/.config/sops/cloudflared`: it is private to you, and it is the one host directory `dev.sh` already mounts into the toolbox (as `/root/.config/sops`), so the toolbox's `sops` can read the credentials JSON without it ever entering the repository tree. ``` CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba' CF_DIR="$HOME/.config/sops/cloudflared" mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR" cf() { podman run --rm -it --user 0 --security-opt label=disable \ -e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; } ``` (Keep `CF_IMG` in step with `platform/cloudflared/deployment.yaml` if Renovate has bumped it.) ### Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard) Universal SSL covers only `keithjackson.dev` and `*.keithjackson.dev`; it does **not** cover the second-level `*.home.keithjackson.dev`. In the `keithjackson.dev` zone: 1. **SSL/TLS → Edge Certificates → Advanced Certificate Manager**: subscribe. 2. Then EITHER **Order an advanced certificate** for exactly `home.keithjackson.dev, *.home.keithjackson.dev` (covers every name at once, independent of proxy state), OR turn on **Total TLS** (issues one certificate per *proxied* hostname as each one appears). The ordered certificate is the more predictable of the two for the later DNS flip; either satisfies the operator decision. 3. **Wait until the certificate shows `Active`** in the Edge Certificates list. Do not do step 6's canary before then — a TLS error at the edge would be the certificate, not the tunnel. 4. Leave **SSL/TLS encryption mode** at *Full (strict)*. (It does not govern tunnel origins — cloudflared verifies the Gateway itself — but it is the safe default for anything else.) Edge settings that would break non-browser clients once traffic moves (check them now, in the zone): - **Security → Bots → Bot Fight Mode: OFF.** It cannot be scoped by hostname or skipped by a WAF rule, and it challenges `git`, `docker`/registry pushes, Nextcloud sync clients and API calls. - **Browser Integrity Check** and the **WAF / managed rules**: make sure neither challenges or blocks non-browser user agents on `gitea.home.keithjackson.dev` (git over HTTPS and the container registry under `/v2/`) or on API paths. If either does, add a WAF **Skip** custom rule for those hostnames/paths rather than turning protection off zone-wide. - **Network → WebSockets: ON** (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC). ### Step 2 — log in and create the tunnel ``` cf tunnel login # Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level # origin certificate. Treat it as a credential. cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s # Prints "Created tunnel home-k3s with id ". The UUID is not secret; note it. cf tunnel list ``` ### Step 3 — put the UUID into the config Do this in your own worktree on a new branch, as CLAUDE.md §0 requires: ``` UUID= sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml grep '^tunnel:' platform/cloudflared/config.yaml ``` ### Step 4 — encrypt the credentials into the Secret, without printing them Run from that worktree. `sops set` edits the already-encrypted document in place; the JSON goes from the mounted file to `sops` inside the container and never reaches the terminal: ``` ./infra/devcontainer/dev.sh run bash -c ' set -euo pipefail cd /work f=/root/.config/sops/cloudflared/tunnel.json v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)" sops set platform/cloudflared/cloudflared-credentials.sops.yaml \ "[\"stringData\"][\"credentials.json\"]" "$v" echo "credentials written" ' ``` (From a worktree, `dev.sh` needs `DEV_KUBECONFIG=/dev/null` and the `.git` mount shim — CLAUDE.md §3. Or run this step from the main checkout and copy the encrypted file across.) Verify — prints only `ok` or fails, never a value: ``` ./infra/devcontainer/dev.sh run bash -c ' cd /work UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml) sops -d --extract "[\"stringData\"][\"credentials.json\"]" \ platform/cloudflared/cloudflared-credentials.sops.yaml \ | jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \ >/dev/null && echo ok' ``` Then store `tunnel.json` **and** `cert.pem` in your password manager, and delete them from disk: ``` rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem" ``` (`cert.pem` is only needed again to inspect (`tunnel info`), create or delete a tunnel. DNS records are external-dns's, never `tunnel route dns`'s.) ### Step 5 — PR it Commit `config.yaml` and the re-encrypted `cloudflared-credentials.sops.yaml`, gate, open a PR. No other wiring is needed: the resource line and both overlay decryption patches already exist. On merge Flux rolls the Deployment (the ConfigMap is content-hashed). That PR touches `**/*.sops.yaml`, a sensitive path in `infra/gitea/approval-policy.json`, so `approval-policy` will block it until you post `/approve-override ` as a PR **review** of type Comment (CLAUDE.md §1). ### Step 6 — verify ``` # Both replicas Ready, on different nodes: infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n cloudflared get pods -o wide' # Logs: "Registered tunnel connection connIndex=0..3" on each replica. infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40' ``` From the workstation: `cf tunnel info home-k3s` (needs `cert.pem` back in `$CF_DIR` temporarily) lists **two connectors**. In Grafana/Prometheus, `sum(cloudflared_tunnel_ha_connections)` is ~8 and both alerts are clear. **Canary — end to end, without touching any real record.** Hand-make ONE proxied CNAME for a throwaway name (Cloudflare dashboard → DNS → Records → Add record): Type CNAME Name tunnel-canary.home Target .cfargotunnel.com Proxy status: Proxied Do NOT use `cloudflared tunnel route dns` for it (see *When to escalate or stop*). The name hits the wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it: ``` curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/ # expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge # certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared # could not reach or verify the Gateway; check the logs and `hubble observe` above. ``` **Then prove a REAL backend answers the tunnel's identity without a 403.** The canary cannot show this: Envoy 404s an unknown host before it resolves any backend, and the `403 Access denied` failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's network identity, so it takes exactly the tunnel's path to the Gateway Service: ``` infra/devcontainer/dev.sh run bash -c ' POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \ -o jsonpath="{.items[0].metadata.name}") kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \ --target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \ --connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"' # expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress # policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404. # A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working. ``` (`kubectl debug` adds an ephemeral container: a cluster mutation, so an operator action. It disappears with the pod on the next rollout.) Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone: `dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing. ## Rotating the tunnel credential Do this whenever the credentials JSON (`TunnelSecret`) or `cert.pem` may have been exposed — a transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this workflow, so rotation is **a new tunnel**: the old credential stays valid until the old tunnel is deleted, which is why the last step is not optional. 1. **Create the new tunnel** (*Enrolling*, step 2's `cf` helper; `cf tunnel login` again if `cert.pem` is not to hand). Use a NEW name — the old one is still taken: ``` cf tunnel list # note the OLD tunnel's id and name cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2 ``` 2. **Swap the UUID** in your own worktree, on a new branch (CLAUDE.md §0): ``` OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml) sed -i "s/^tunnel: ${OLD}$/tunnel: /" platform/cloudflared/config.yaml # Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit; # the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree. sed -i "s/${OLD}\.cfargotunnel\.com/.cfargotunnel.com/" platform/gateway/gateway.yaml ``` 3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never `sops -d` to the terminal. 4. **Merge** — gate, PR, `/approve-override` by the operator (it touches `**/*.sops.yaml`), merge. Flux rolls the Deployment because the content-hashed ConfigMap changed. 5. **Verify** with *Enrolling* step 6: both replicas Ready, `Registered tunnel connection` in each replica's log, `cf tunnel info home-k3s-2` lists two connectors, and `cf tunnel info ` lists **none**. 6. **Only then delete the old tunnel**, which revokes the exposed credential: ``` cf tunnel cleanup # drops any stale connector records first cf tunnel delete cf tunnel list # the old tunnel is gone ``` Since Phase 2, records point at `.cfargotunnel.com`: confirm external-dns has repointed every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those hostnames go dark. Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk. ## Rollback - **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and everything in it. No user traffic is affected either way. Optionally `cf tunnel delete home-k3s` (needs `cert.pem`) to retire the credential. - **Emergency stop (operator, cluster mutation):** stop the connectors without waiting for a PR: ``` infra/devcontainer/dev.sh run bash -c \ 'flux --context k3s-prod suspend kustomization cloudflared && kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0' ``` Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then `flux resume kustomization cloudflared`. - **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel` (Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard** (`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the `pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand: external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.