home-network-k3s-impl/runbooks/cloudflare-tunnel.md

cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts

Alerts that link here: CloudflaredTunnelDown (no replica has a live edge connection, or none is being scraped) and CloudflaredReplicasDegraded (fewer than 2 replicas Available).

Component: platform/cloudflared/ (Flux Kustomization cloudflared, clusters/base/cloudflared.yaml). Design record: decisions/2026-11-08-cloudflare-tunnel.md.

Every step below is an operator step. An agent must not mint the tunnel credential, run flux reconcile, or mutate the cluster (CLAUDE.md §5, §6).

What this means here

cloudflared is an outbound-only connector: two replicas dial the Cloudflare edge on 7844 (QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway (https://cilium-gateway-shared.gateway.svc.cluster.local:443, SNI = the request's Host, so the Gateway's existing wildcard-home-tls Let's Encrypt certificate is what cloudflared validates). The tunnel is locally managed: its ingress rules are platform/cloudflared/config.yaml, in git, not in the Cloudflare dashboard.

The operator decisions it encodes (2026-11-08):

1. Public hostnames stay *.home.keithjackson.dev and the apex; edge certificates come from Cloudflare ACM + Total TLS.
2. Plex is not on the tunnel (plex, plex-vip → 404).
3. jackett, pihole, semaphore, qbittorrent, radarr, sonarr and headlamp stay LAN/VPN-only, as do the L4 VIPs unifi and crafty-mc (all → 404 in cloudflared, and the LAN-only Gateway backends are also refused by the egress policy). Headlamp is an operator decision of 2026-11-08; the other six are the catalog's exposure: internal services.
4. The Kubernetes API is never reachable through the tunnel. No rule targets it and the egress policy admits neither the kube-apiserver nor the host entity. The L0 check cloudflared-ingress-rules fails the gate if that ever changes.

Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to <UUID>.cfargotunnel.com. Phase 1 created no such record.

Phase 2 — DONE (2026-11-08, branch feat/dns-via-cloudflare-tunnel). external-dns now publishes every gateway name as a proxied CNAME to the tunnel: the shared Gateway's external-dns.alpha.kubernetes.io/target is <UUID>.cfargotunnel.com and the HelmRelease sets --cloudflare-proxied (see platform/externaldns/README.md). Plex's web UI and Headlamp are disowned (controller: none), so their public records are deleted; unifi and crafty-mc stay DNS-only A records (cloudflare-proxied: "false"); wan. stays for WireGuard. The L0 check internal-services-publish-no-public-dns (9) pins all of this, including that the Gateway target's UUID equals tunnel: in config.yaml. So from Phase 2 on, these alerts firing DOES mean public users are down — every public HTTP(S) name depends on the connectors.

After the Phase 2 merge — operator step (router)

Once dig +short @1.1.1.1 <public name> returns Cloudflare anycast addresses for the public names (and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:

1. Close the router's WAN tcp/80 and tcp/443 port-forwards to the Gateway VIP.
2. Keep WAN udp/51820 (WireGuard → wan.home.keithjackson.dev) and WAN tcp/32400 (Plex remote streaming via plex.tv; runbooks/plex-remote-access.md).
3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and curl -m5 https://<WAN IP>/ now times out instead of answering.

Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and CloudflaredTunnelDown fires. That is expected, and it is contained: the cloudflared Kustomization is wait: false and nothing depends on it.

First commands

# Are the pods running, and on different nodes?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide'

# What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50'

# Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization cloudflared;
   kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \
     -o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"'

Never print the Secret's values. ENC[ in the decrypted Secret means the overlay's decryption patch is missing; see causes below.

Likely causes, most likely first

Read the FIRST error line, not the last. When cloudflared cannot use the credentials file it falls back to looking for an origin certificate, and the error it ends on — Cannot determine default origin certificate path / error parsing tunnel ID: Error locating origin cert — is a red herring whatever the real cause was. The credentials file is the whole of what this Deployment needs; no cert.pem ever belongs in the pod.

1. permission denied on credentials.json → the "origin cert" error is a red herring. Logs: couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission denied, then the origin-cert lines. The container (uid 65532) cannot read the Secret file: the pod securityContext.fsGroup: 65532 is missing, so the 0440 file is root:root. Fix it in git (platform/cloudflared/deployment.yaml); the L0 check cloudflared-ingress-rules, assertion (7), should already have refused that. Do NOT loosen defaultMode to 0444. (This is what CrashLooped every replica on 2026-11-08.)
2. The tunnel has not been enrolled yet (Phase 1, steps 1-5 below not done). Logs show an error about the tunnel REPLACE-ME-TUNNEL-UUID (and the origin-cert red herring). Do steps 1-5.
3. UUID and credentials disagree — config.yaml tunnel: is not the TunnelID in the credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
4. Credentials still encrypted in the cluster — the cloudflared decryption patch is missing from clusters/prod/kustomization.yaml. The L0 check guards this, so this means someone bypassed it.
5. Egress blocked — a policy change, or the edge moved off 7844. Look for drops:

   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
        hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
   

   (ds/cilium picks one node's agent; repeat per node if empty.)
6. The tunnel was deleted or its secret rotated in Cloudflare — cloudflared tunnel list (step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
7. CloudflaredReplicasDegraded only: one replica cannot schedule (the spread is hard, one per node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.

When to escalate or stop

• Stop if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently overridden by config.yaml. Use the emergency stop under Rollback, then fix in git.
• Never run cloudflared tunnel route dns — not for a real hostname, not for a test one. external-dns owns the home.keithjackson.dev records (platform/externaldns, policy: sync); a record created behind its back either fights it or is deleted by it, and a real hostname routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns (Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
• Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.

---

Enrolling the tunnel (once)

cloudflared is not in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image the Deployment uses, under podman — no host install. The working directory is ~/.config/sops/cloudflared: it is private to you, and it is the one host directory dev.sh already mounts into the toolbox (as /root/.config/sops), so the toolbox's sops can read the credentials JSON without it ever entering the repository tree.

CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba'
CF_DIR="$HOME/.config/sops/cloudflared"
mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR"
cf() { podman run --rm -it --user 0 --security-opt label=disable \
         -e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; }

(Keep CF_IMG in step with platform/cloudflared/deployment.yaml if Renovate has bumped it.)

Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard)

Universal SSL covers only keithjackson.dev and *.keithjackson.dev; it does not cover the second-level *.home.keithjackson.dev. In the keithjackson.dev zone:

1. SSL/TLS → Edge Certificates → Advanced Certificate Manager: subscribe.
2. Then EITHER Order an advanced certificate for exactly home.keithjackson.dev, *.home.keithjackson.dev (covers every name at once, independent of proxy state), OR turn on Total TLS (issues one certificate per proxied hostname as each one appears). The ordered certificate is the more predictable of the two for the later DNS flip; either satisfies the operator decision.
3. Wait until the certificate shows Active in the Edge Certificates list. Do not do step 6's canary before then — a TLS error at the edge would be the certificate, not the tunnel.
4. Leave SSL/TLS encryption mode at Full (strict). (It does not govern tunnel origins — cloudflared verifies the Gateway itself — but it is the safe default for anything else.)

Edge settings that would break non-browser clients once traffic moves (check them now, in the zone):

• Security → Bots → Bot Fight Mode: OFF. It cannot be scoped by hostname or skipped by a WAF rule, and it challenges git, docker/registry pushes, Nextcloud sync clients and API calls.
• Browser Integrity Check and the WAF / managed rules: make sure neither challenges or blocks non-browser user agents on gitea.home.keithjackson.dev (git over HTTPS and the container registry under /v2/) or on API paths. If either does, add a WAF Skip custom rule for those hostnames/paths rather than turning protection off zone-wide.
• Network → WebSockets: ON (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC).

Step 2 — log in and create the tunnel

cf tunnel login
# Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level
# origin certificate. Treat it as a credential.

cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s
# Prints "Created tunnel home-k3s with id <UUID>". The UUID is not secret; note it.
cf tunnel list

Step 3 — put the UUID into the config

Do this in your own worktree on a new branch, as CLAUDE.md §0 requires:

UUID=<the id from step 2>
sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml
grep '^tunnel:' platform/cloudflared/config.yaml

Step 4 — encrypt the credentials into the Secret, without printing them

Run from that worktree. sops set edits the already-encrypted document in place; the JSON goes from the mounted file to sops inside the container and never reaches the terminal:

./infra/devcontainer/dev.sh run bash -c '
  set -euo pipefail
  cd /work
  f=/root/.config/sops/cloudflared/tunnel.json
  v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)"
  sops set platform/cloudflared/cloudflared-credentials.sops.yaml \
    "[\"stringData\"][\"credentials.json\"]" "$v"
  echo "credentials written"
'

(From a worktree, dev.sh needs DEV_KUBECONFIG=/dev/null and the .git mount shim — CLAUDE.md §3. Or run this step from the main checkout and copy the encrypted file across.)

Verify — prints only ok or fails, never a value:

./infra/devcontainer/dev.sh run bash -c '
  cd /work
  UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml)
  sops -d --extract "[\"stringData\"][\"credentials.json\"]" \
      platform/cloudflared/cloudflared-credentials.sops.yaml \
    | jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \
      >/dev/null && echo ok'

Then store tunnel.json and cert.pem in your password manager, and delete them from disk:

rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem"

(cert.pem is only needed again to inspect (tunnel info), create or delete a tunnel. DNS records are external-dns's, never tunnel route dns's.)

Step 5 — PR it

Commit config.yaml and the re-encrypted cloudflared-credentials.sops.yaml, gate, open a PR. No other wiring is needed: the resource line and both overlay decryption patches already exist. On merge Flux rolls the Deployment (the ConfigMap is content-hashed).

That PR touches **/*.sops.yaml, a sensitive path in infra/gitea/approval-policy.json, so approval-policy will block it until you post /approve-override <reason> as a PR review of type Comment (CLAUDE.md §1).

Step 6 — verify

# Both replicas Ready, on different nodes:
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get pods -o wide'

# Logs: "Registered tunnel connection connIndex=0..3" on each replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40'

From the workstation: cf tunnel info home-k3s (needs cert.pem back in $CF_DIR temporarily) lists two connectors. In Grafana/Prometheus, sum(cloudflared_tunnel_ha_connections) is ~8 and both alerts are clear.

Canary — end to end, without touching any real record. Hand-make ONE proxied CNAME for a throwaway name (Cloudflare dashboard → DNS → Records → Add record):

Type CNAME   Name tunnel-canary.home   Target <UUID>.cfargotunnel.com   Proxy status: Proxied

Do NOT use cloudflared tunnel route dns for it (see When to escalate or stop). The name hits the wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it:

curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/
# expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge
# certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared
# could not reach or verify the Gateway; check the logs and `hubble observe` above.

Then prove a REAL backend answers the tunnel's identity without a 403. The canary cannot show this: Envoy 404s an unknown host before it resolves any backend, and the 403 Access denied failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's network identity, so it takes exactly the tunnel's path to the Gateway Service:

infra/devcontainer/dev.sh run bash -c '
  POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \
          -o jsonpath="{.items[0].metadata.name}")
  kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \
    --target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do
      printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \
        --connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"'
# expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress
# policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404.
# A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working.

(kubectl debug adds an ephemeral container: a cluster mutation, so an operator action. It disappears with the pod on the next rollout.)

Finally delete the tunnel-canary CNAME in the dashboard and confirm it is gone: dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1 returns nothing.

Rotating the tunnel credential

Do this whenever the credentials JSON (TunnelSecret) or cert.pem may have been exposed — a transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this workflow, so rotation is a new tunnel: the old credential stays valid until the old tunnel is deleted, which is why the last step is not optional.

1. Create the new tunnel (Enrolling, step 2's cf helper; cf tunnel login again if cert.pem is not to hand). Use a NEW name — the old one is still taken:

   cf tunnel list                       # note the OLD tunnel's id and name
   cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2
   

2. Swap the UUID in your own worktree, on a new branch (CLAUDE.md §0):

   OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
   sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
   # Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
   # the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
   sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
   

3. Swap the encrypted credentials with Enrolling step 4 — the same sops set, then the same ok-only verify, which now checks the JSON's TunnelID against the NEW UUID. Never sops -d to the terminal.
4. Merge — gate, PR, /approve-override by the operator (it touches **/*.sops.yaml), merge. Flux rolls the Deployment because the content-hashed ConfigMap changed.
5. Verify with Enrolling step 6: both replicas Ready, Registered tunnel connection in each replica's log, cf tunnel info home-k3s-2 lists two connectors, and cf tunnel info <old name> lists none.
6. Only then delete the old tunnel, which revokes the exposed credential:

   cf tunnel cleanup <old name>   # drops any stale connector records first
   cf tunnel delete <old name>
   cf tunnel list                 # the old tunnel is gone
   

   Since Phase 2, records point at <UUID>.cfargotunnel.com: confirm external-dns has repointed every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those hostnames go dark. Then replace tunnel.json/cert.pem in the password manager and delete them from disk.

Rollback

• Phase 1 (no DNS on the tunnel): revert the PR. Flux prunes the cloudflared namespace and everything in it. No user traffic is affected either way. Optionally cf tunnel delete home-k3s (needs cert.pem) to retire the credential.
• Emergency stop (operator, cluster mutation): stop the connectors without waiting for a PR:

  infra/devcontainer/dev.sh run bash -c \
    'flux --context k3s-prod suspend kustomization cloudflared &&
     kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
  

  Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then flux resume kustomization cloudflared.
• After the DNS flip (Phase 2): the path back is a revert PR of feat/dns-via-cloudflare-tunnel (Gateway target back to wan.${BASE_DOMAIN}, drop --cloudflare-proxied) — external-dns then rewrites every record DNS-only to wan.. That only works if the router's WAN tcp/80+443 forwards are re-opened first (they are closed after Phase 2). Never edit the records by hand: external-dns owns them and policy: sync reverts a hand edit. Revoking the tunnel credential (cf tunnel delete, or rotating it in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.