home-network-k3s-impl/runbooks/cloudflare-tunnel.md

cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts

Alerts that link here: CloudflaredTunnelDown (no replica has a live edge connection, or none is being scraped) and CloudflaredReplicasDegraded (fewer than 2 replicas Available).

Component: platform/cloudflared/ (Flux Kustomization cloudflared, clusters/base/cloudflared.yaml). Design record: decisions/2026-11-08-cloudflare-tunnel.md.

Every step below is an operator step. An agent must not mint the tunnel credential, run flux reconcile, or mutate the cluster (CLAUDE.md §5, §6).

What this means here

cloudflared is an outbound-only connector: two replicas dial the Cloudflare edge on 7844 (QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway (https://cilium-gateway-shared.gateway.svc.cluster.local:443, SNI = the request's Host, so the Gateway's existing wildcard-home-tls Let's Encrypt certificate is what cloudflared validates). The tunnel is locally managed: its ingress rules are platform/cloudflared/config.yaml, in git, not in the Cloudflare dashboard.

The operator decisions it encodes (2026-11-08):

1. Public hostnames stay *.home.keithjackson.dev and the apex; edge certificates come from Cloudflare ACM + Total TLS.
2. Plex is not on the tunnel (plex, plex-vip → 404).
3. jackett, pihole, semaphore, qbittorrent, radarr, sonarr and headlamp stay LAN/VPN-only, as do the L4 VIPs unifi and crafty-mc (all → 404 in cloudflared, and the LAN-only Gateway backends are also refused by the egress policy). Headlamp is an operator decision of 2026-11-08; the other six are the catalog's exposure: internal services.
4. The Kubernetes API is never reachable through the tunnel. No rule targets it and the egress policy admits neither the kube-apiserver nor the host entity. The L0 check cloudflared-ingress-rules fails the gate if that ever changes.

Nothing goes through the tunnel until DNS points at it. Cloudflare routes a hostname through a tunnel only when its record is a CNAME to <UUID>.cfargotunnel.com. Phase 1 (this component) creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts firing means "the tunnel is not ready", not "users are down".

Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and CloudflaredTunnelDown fires. That is expected, and it is contained: the cloudflared Kustomization is wait: false and nothing depends on it.

First commands

# Are the pods running, and on different nodes?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide'

# What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50'

# Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization cloudflared;
   kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \
     -o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"'

Never print the Secret's values. ENC[ in the decrypted Secret means the overlay's decryption patch is missing; see causes below.

Likely causes, most likely first

1. The tunnel has not been enrolled yet (Phase 1, steps 1-5 below not done). Logs show an error about the tunnel REPLACE-ME-TUNNEL-UUID or a missing origin certificate. Do steps 1-5.
2. UUID and credentials disagree — config.yaml tunnel: is not the TunnelID in the credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
3. Credentials still encrypted in the cluster — the cloudflared decryption patch is missing from clusters/prod/kustomization.yaml. The L0 check guards this, so this means someone bypassed it.
4. Egress blocked — a policy change, or the edge moved off 7844. Look for drops:

   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
        hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
   

   (ds/cilium picks one node's agent; repeat per node if empty.)
5. The tunnel was deleted or its secret rotated in Cloudflare — cloudflared tunnel list (step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
6. CloudflaredReplicasDegraded only: one replica cannot schedule (the spread is hard, one per node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.

When to escalate or stop

• Stop if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently overridden by config.yaml. Use the emergency stop under Rollback, then fix in git.
• Never run cloudflared tunnel route dns — not for a real hostname, not for a test one. external-dns owns the home.keithjackson.dev records (platform/externaldns, policy: sync); a record created behind its back either fights it or is deleted by it, and a real hostname routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2 change. The only record this runbook creates is the hand-made canary in step 6, and you delete it.
• Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.

---

Enrolling the tunnel (once)

cloudflared is not in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image the Deployment uses, under podman — no host install. The working directory is ~/.config/sops/cloudflared: it is private to you, and it is the one host directory dev.sh already mounts into the toolbox (as /root/.config/sops), so the toolbox's sops can read the credentials JSON without it ever entering the repository tree.

CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba'
CF_DIR="$HOME/.config/sops/cloudflared"
mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR"
cf() { podman run --rm -it --user 0 --security-opt label=disable \
         -e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; }

(Keep CF_IMG in step with platform/cloudflared/deployment.yaml if Renovate has bumped it.)

Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard)

Universal SSL covers only keithjackson.dev and *.keithjackson.dev; it does not cover the second-level *.home.keithjackson.dev. In the keithjackson.dev zone:

1. SSL/TLS → Edge Certificates → Advanced Certificate Manager: subscribe.
2. Then EITHER Order an advanced certificate for exactly home.keithjackson.dev, *.home.keithjackson.dev (covers every name at once, independent of proxy state), OR turn on Total TLS (issues one certificate per proxied hostname as each one appears). The ordered certificate is the more predictable of the two for the later DNS flip; either satisfies the operator decision.
3. Wait until the certificate shows Active in the Edge Certificates list. Do not do step 6's canary before then — a TLS error at the edge would be the certificate, not the tunnel.
4. Leave SSL/TLS encryption mode at Full (strict). (It does not govern tunnel origins — cloudflared verifies the Gateway itself — but it is the safe default for anything else.)

Edge settings that would break non-browser clients once traffic moves (check them now, in the zone):

• Security → Bots → Bot Fight Mode: OFF. It cannot be scoped by hostname or skipped by a WAF rule, and it challenges git, docker/registry pushes, Nextcloud sync clients and API calls.
• Browser Integrity Check and the WAF / managed rules: make sure neither challenges or blocks non-browser user agents on gitea.home.keithjackson.dev (git over HTTPS and the container registry under /v2/) or on API paths. If either does, add a WAF Skip custom rule for those hostnames/paths rather than turning protection off zone-wide.
• Network → WebSockets: ON (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC).

Step 2 — log in and create the tunnel

cf tunnel login
# Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level
# origin certificate. Treat it as a credential.

cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s
# Prints "Created tunnel home-k3s with id <UUID>". The UUID is not secret; note it.
cf tunnel list

Step 3 — put the UUID into the config

Do this in your own worktree on a new branch, as CLAUDE.md §0 requires:

UUID=<the id from step 2>
sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml
grep '^tunnel:' platform/cloudflared/config.yaml

Step 4 — encrypt the credentials into the Secret, without printing them

Run from that worktree. sops set edits the already-encrypted document in place; the JSON goes from the mounted file to sops inside the container and never reaches the terminal:

./infra/devcontainer/dev.sh run bash -c '
  set -euo pipefail
  cd /work
  f=/root/.config/sops/cloudflared/tunnel.json
  v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)"
  sops set platform/cloudflared/cloudflared-credentials.sops.yaml \
    "[\"stringData\"][\"credentials.json\"]" "$v"
  echo "credentials written"
'

(From a worktree, dev.sh needs DEV_KUBECONFIG=/dev/null and the .git mount shim — CLAUDE.md §3. Or run this step from the main checkout and copy the encrypted file across.)

Verify — prints only ok or fails, never a value:

./infra/devcontainer/dev.sh run bash -c '
  cd /work
  UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml)
  sops -d --extract "[\"stringData\"][\"credentials.json\"]" \
      platform/cloudflared/cloudflared-credentials.sops.yaml \
    | jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \
      >/dev/null && echo ok'

Then store tunnel.json and cert.pem in your password manager, and delete them from disk:

rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem"

(cert.pem is only needed again to inspect (tunnel info), create or delete a tunnel. DNS records are external-dns's, never tunnel route dns's.)

Step 5 — PR it

Commit config.yaml and the re-encrypted cloudflared-credentials.sops.yaml, gate, open a PR. No other wiring is needed: the resource line and both overlay decryption patches already exist. On merge Flux rolls the Deployment (the ConfigMap is content-hashed).

That PR touches **/*.sops.yaml, a sensitive path in infra/gitea/approval-policy.json, so approval-policy will block it until you post /approve-override <reason> as a PR review of type Comment (CLAUDE.md §1).

Step 6 — verify

# Both replicas Ready, on different nodes:
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get pods -o wide'

# Logs: "Registered tunnel connection connIndex=0..3" on each replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40'

From the workstation: cf tunnel info home-k3s (needs cert.pem back in $CF_DIR temporarily) lists two connectors. In Grafana/Prometheus, sum(cloudflared_tunnel_ha_connections) is ~8 and both alerts are clear.

Canary — end to end, without touching any real record. Hand-make ONE proxied CNAME for a throwaway name (Cloudflare dashboard → DNS → Records → Add record):

Type CNAME   Name tunnel-canary.home   Target <UUID>.cfargotunnel.com   Proxy status: Proxied

Do NOT use cloudflared tunnel route dns for it (see When to escalate or stop). The name hits the wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it:

curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/
# expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge
# certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared
# could not reach or verify the Gateway; check the logs and `hubble observe` above.

Then prove a REAL backend answers the tunnel's identity without a 403. The canary cannot show this: Envoy 404s an unknown host before it resolves any backend, and the 403 Access denied failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's network identity, so it takes exactly the tunnel's path to the Gateway Service:

infra/devcontainer/dev.sh run bash -c '
  POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \
          -o jsonpath="{.items[0].metadata.name}")
  kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \
    --target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do
      printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \
        --connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"'
# expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress
# policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404.
# A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working.

(kubectl debug adds an ephemeral container: a cluster mutation, so an operator action. It disappears with the pod on the next rollout.)

Finally delete the tunnel-canary CNAME in the dashboard and confirm it is gone: dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1 returns nothing.

Rollback

• Phase 1 (no DNS on the tunnel): revert the PR. Flux prunes the cloudflared namespace and everything in it. No user traffic is affected either way. Optionally cf tunnel delete home-k3s (needs cert.pem) to retire the credential.
• Emergency stop (operator, cluster mutation): stop the connectors without waiting for a PR:

  infra/devcontainer/dev.sh run bash -c \
    'flux --context k3s-prod suspend kustomization cloudflared &&
     kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
  

  Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then flux resume kustomization cloudflared.
• After the DNS flip: the fast path back is DNS, not the tunnel — restore the previous records (that change's own runbook). Revoking the tunnel credential (cf tunnel delete, or rotating it in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.