home-network-k3s-impl/runbooks/cloudflare-tunnel.md

# cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts

Alerts that link here: **`CloudflaredTunnelDown`** (no replica has a live edge connection, or none
is being scraped) and **`CloudflaredReplicasDegraded`** (fewer than 2 replicas Available).

Component: `platform/cloudflared/` (Flux Kustomization `cloudflared`, `clusters/base/cloudflared.yaml`).
Design record: `decisions/2026-11-08-cloudflare-tunnel.md`.

Every step below is an **operator** step. An agent must not mint the tunnel credential, run
`flux reconcile`, or mutate the cluster (CLAUDE.md §5, §6).

## What this means here

cloudflared is an **outbound-only** connector: two replicas dial the Cloudflare edge on 7844
(QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway
(`https://cilium-gateway-shared.gateway.svc.cluster.local:443`, SNI = the request's Host, so the
Gateway's existing `wildcard-home-tls` Let's Encrypt certificate is what cloudflared validates).
The tunnel is **locally managed**: its ingress rules are `platform/cloudflared/config.yaml`, in git,
not in the Cloudflare dashboard.

The operator decisions it encodes (2026-11-08):

1. Public hostnames stay `*.home.keithjackson.dev` and the apex; edge certificates come from
   Cloudflare **ACM + Total TLS**.
2. **Plex is not on the tunnel** (`plex`, `plex-vip` → 404).
3. **jackett, pihole, semaphore, qbittorrent, radarr, sonarr** and **headlamp** stay LAN/VPN-only,
   as do the L4 VIPs **unifi** and **crafty-mc** (all → 404 in cloudflared, and the LAN-only
   Gateway backends are also refused by the egress policy). Headlamp is an operator decision of
   2026-11-08; the other six are the catalog's `exposure: internal` services.
4. **The Kubernetes API is never reachable through the tunnel.** No rule targets it and the egress
   policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
   `cloudflared-ingress-rules` fails the gate if that ever changes.

**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a
tunnel only when its record is a CNAME to `<UUID>.cfargotunnel.com`. Phase 1 (this component)
creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts
firing means "the tunnel is not ready", not "users are down".

Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
`CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
is `wait: false` and nothing depends on it.

## First commands

```
# Are the pods running, and on different nodes?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide'

# What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50'

# Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization cloudflared;
   kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \
     -o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"'
```

Never print the Secret's values. `ENC[` in the decrypted Secret means the overlay's decryption
patch is missing; see causes below.

## Likely causes, most likely first

1. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
   error about the tunnel `REPLACE-ME-TUNNEL-UUID` or a missing origin certificate. Do steps 1-5.
2. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
   credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
3. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
   from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed
   it.
4. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
   ```
   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
        hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
   ```
   (`ds/cilium` picks one node's agent; repeat per node if empty.)
5. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
   (step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
6. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
   node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.

## When to escalate or stop

- **Stop** if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not
  hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently
  overridden by `config.yaml`. Use the emergency stop under *Rollback*, then fix in git.
- **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
  **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
  a record created behind its back either fights it or is deleted by it, and a real hostname
  routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2
  change. The only record this runbook creates is the hand-made canary in step 6, and you delete it.
- Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
  Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.

---

## Enrolling the tunnel (once)

cloudflared is **not** in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image
the Deployment uses, under podman — no host install. The working directory is
`~/.config/sops/cloudflared`: it is private to you, and it is the one host directory `dev.sh`
already mounts into the toolbox (as `/root/.config/sops`), so the toolbox's `sops` can read the
credentials JSON without it ever entering the repository tree.

```
CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba'
CF_DIR="$HOME/.config/sops/cloudflared"
mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR"
cf() { podman run --rm -it --user 0 --security-opt label=disable \
         -e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; }
```

(Keep `CF_IMG` in step with `platform/cloudflared/deployment.yaml` if Renovate has bumped it.)

### Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard)

Universal SSL covers only `keithjackson.dev` and `*.keithjackson.dev`; it does **not** cover the
second-level `*.home.keithjackson.dev`. In the `keithjackson.dev` zone:

1. **SSL/TLS → Edge Certificates → Advanced Certificate Manager**: subscribe.
2. Then EITHER **Order an advanced certificate** for exactly `home.keithjackson.dev,
   *.home.keithjackson.dev` (covers every name at once, independent of proxy state), OR turn on
   **Total TLS** (issues one certificate per *proxied* hostname as each one appears). The ordered
   certificate is the more predictable of the two for the later DNS flip; either satisfies the
   operator decision.
3. **Wait until the certificate shows `Active`** in the Edge Certificates list. Do not do step 6's
   canary before then — a TLS error at the edge would be the certificate, not the tunnel.
4. Leave **SSL/TLS encryption mode** at *Full (strict)*. (It does not govern tunnel origins —
   cloudflared verifies the Gateway itself — but it is the safe default for anything else.)

Edge settings that would break non-browser clients once traffic moves (check them now, in the zone):

- **Security → Bots → Bot Fight Mode: OFF.** It cannot be scoped by hostname or skipped by a WAF
  rule, and it challenges `git`, `docker`/registry pushes, Nextcloud sync clients and API calls.
- **Browser Integrity Check** and the **WAF / managed rules**: make sure neither challenges or
  blocks non-browser user agents on `gitea.home.keithjackson.dev` (git over HTTPS and the container
  registry under `/v2/`) or on API paths. If either does, add a WAF **Skip** custom rule for those
  hostnames/paths rather than turning protection off zone-wide.
- **Network → WebSockets: ON** (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC).

### Step 2 — log in and create the tunnel

```
cf tunnel login
# Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level
# origin certificate. Treat it as a credential.

cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s
# Prints "Created tunnel home-k3s with id <UUID>". The UUID is not secret; note it.
cf tunnel list
```

### Step 3 — put the UUID into the config

Do this in your own worktree on a new branch, as CLAUDE.md §0 requires:

```
UUID=<the id from step 2>
sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml
grep '^tunnel:' platform/cloudflared/config.yaml
```

### Step 4 — encrypt the credentials into the Secret, without printing them

Run from that worktree. `sops set` edits the already-encrypted document in place; the JSON goes
from the mounted file to `sops` inside the container and never reaches the terminal:

```
./infra/devcontainer/dev.sh run bash -c '
  set -euo pipefail
  cd /work
  f=/root/.config/sops/cloudflared/tunnel.json
  v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)"
  sops set platform/cloudflared/cloudflared-credentials.sops.yaml \
    "[\"stringData\"][\"credentials.json\"]" "$v"
  echo "credentials written"
'
```

(From a worktree, `dev.sh` needs `DEV_KUBECONFIG=/dev/null` and the `.git` mount shim — CLAUDE.md §3.
Or run this step from the main checkout and copy the encrypted file across.)

Verify — prints only `ok` or fails, never a value:

```
./infra/devcontainer/dev.sh run bash -c '
  cd /work
  UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml)
  sops -d --extract "[\"stringData\"][\"credentials.json\"]" \
      platform/cloudflared/cloudflared-credentials.sops.yaml \
    | jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \
      >/dev/null && echo ok'
```

Then store `tunnel.json` **and** `cert.pem` in your password manager, and delete them from disk:

```
rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem"
```

(`cert.pem` is only needed again to inspect (`tunnel info`), create or delete a tunnel. DNS records are
external-dns's, never `tunnel route dns`'s.)

### Step 5 — PR it

Commit `config.yaml` and the re-encrypted `cloudflared-credentials.sops.yaml`, gate, open a PR. No
other wiring is needed: the resource line and both overlay decryption patches already exist. On
merge Flux rolls the Deployment (the ConfigMap is content-hashed).

That PR touches `**/*.sops.yaml`, a sensitive path in `infra/gitea/approval-policy.json`, so
`approval-policy` will block it until you post `/approve-override <reason>` as a PR **review** of
type Comment (CLAUDE.md §1).

### Step 6 — verify

```
# Both replicas Ready, on different nodes:
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get pods -o wide'

# Logs: "Registered tunnel connection connIndex=0..3" on each replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40'
```

From the workstation: `cf tunnel info home-k3s` (needs `cert.pem` back in `$CF_DIR` temporarily)
lists **two connectors**. In Grafana/Prometheus, `sum(cloudflared_tunnel_ha_connections)` is ~8 and
both alerts are clear.

**Canary — end to end, without touching any real record.** Hand-make ONE proxied CNAME for a
throwaway name (Cloudflare dashboard → DNS → Records → Add record):

    Type CNAME   Name tunnel-canary.home   Target <UUID>.cfargotunnel.com   Proxy status: Proxied

Do NOT use `cloudflared tunnel route dns` for it (see *When to escalate or stop*). The name hits the
wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it:

```
curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/
# expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge
# certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared
# could not reach or verify the Gateway; check the logs and `hubble observe` above.
```

**Then prove a REAL backend answers the tunnel's identity without a 403.** The canary cannot show
this: Envoy 404s an unknown host before it resolves any backend, and the `403 Access denied`
failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only
happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's
network identity, so it takes exactly the tunnel's path to the Gateway Service:

```
infra/devcontainer/dev.sh run bash -c '
  POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \
          -o jsonpath="{.items[0].metadata.name}")
  kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \
    --target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do
      printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \
        --connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"'
# expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress
# policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404.
# A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working.
```

(`kubectl debug` adds an ephemeral container: a cluster mutation, so an operator action. It
disappears with the pod on the next rollout.)

Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone:
`dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing.

## Rollback

- **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and
  everything in it. No user traffic is affected either way. Optionally `cf tunnel delete home-k3s`
  (needs `cert.pem`) to retire the credential.
- **Emergency stop (operator, cluster mutation):** stop the connectors without waiting for a PR:
  ```
  infra/devcontainer/dev.sh run bash -c \
    'flux --context k3s-prod suspend kustomization cloudflared &&
     kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
  ```
  Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
  `flux resume kustomization cloudflared`.
- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records
  (that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it
  in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.