home-network-k3s-impl/runbooks/cloudflare-tunnel.md

# cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts

Alerts that link here: **`CloudflaredTunnelDown`** (no replica has a live edge connection, or none
is being scraped) and **`CloudflaredReplicasDegraded`** (fewer than 2 replicas Available).

Component: `platform/cloudflared/` (Flux Kustomization `cloudflared`, `clusters/base/cloudflared.yaml`).
Design record: `decisions/2026-11-08-cloudflare-tunnel.md`.

Every step below is an **operator** step. An agent must not mint the tunnel credential, run
`flux reconcile`, or mutate the cluster (CLAUDE.md §5, §6).

## What this means here

cloudflared is an **outbound-only** connector: two replicas dial the Cloudflare edge on 7844
(QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway
(`https://cilium-gateway-shared.gateway.svc.cluster.local:443`, SNI = the request's Host, so the
Gateway's existing `wildcard-home-tls` Let's Encrypt certificate is what cloudflared validates).
The tunnel is **locally managed**: its ingress rules are `platform/cloudflared/config.yaml`, in git,
not in the Cloudflare dashboard.

The operator decisions it encodes (2026-11-08):

1. Public hostnames stay `*.home.keithjackson.dev` and the apex; edge certificates come from
   Cloudflare **ACM + Total TLS**.
2. **Plex is not on the tunnel** (`plex`, `plex-vip` → 404).
3. **jackett, pihole, semaphore, qbittorrent, radarr, sonarr** and **headlamp** stay LAN/VPN-only,
   as do the L4 VIPs **unifi** and **crafty-mc** (all → 404 in cloudflared, and the LAN-only
   Gateway backends are also refused by the egress policy). Headlamp is an operator decision of
   2026-11-08; the other six are the catalog's `exposure: internal` services.
4. **The Kubernetes API is never reachable through the tunnel.** No rule targets it and the egress
   policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
   `cloudflared-ingress-rules` fails the gate if that ever changes.

**Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to
`<UUID>.cfargotunnel.com`.** Phase 1 created no such record.

**Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes
every gateway name as a proxied CNAME to the tunnel: the shared Gateway's
`external-dns.alpha.kubernetes.io/target` is `<UUID>.cfargotunnel.com` and the HelmRelease sets
`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are
disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only
A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check
`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's
UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public
users are down** — every public HTTP(S) name depends on the connectors.

### After the Phase 2 merge — operator step (router)

Once `dig +short @1.1.1.1 <public name>` returns Cloudflare anycast addresses for the public names
(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:

1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP.
2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400**
   (Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`).
3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and
   `curl -m5 https://<WAN IP>/` now times out instead of answering.

Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
`CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
is `wait: false` and nothing depends on it.

## First commands

```
# Are the pods running, and on different nodes?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide'

# What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50'

# Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)?
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization cloudflared;
   kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \
     -o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"'
```

Never print the Secret's values. `ENC[` in the decrypted Secret means the overlay's decryption
patch is missing; see causes below.

## Likely causes, most likely first

**Read the FIRST error line, not the last.** When cloudflared cannot use the credentials file it
falls back to looking for an origin certificate, and the error it ends on —
`Cannot determine default origin certificate path` / `error parsing tunnel ID: Error locating
origin cert` — is a **red herring** whatever the real cause was. The credentials file is the
whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.

1. **`permission denied` on `credentials.json` → the "origin cert" error is a red herring.** Logs:
   `couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission
   denied`, then the origin-cert lines. The container (uid 65532) cannot read the Secret file:
   the pod `securityContext.fsGroup: 65532` is missing, so the 0440 file is `root:root`. Fix it in
   git (`platform/cloudflared/deployment.yaml`); the L0 check `cloudflared-ingress-rules`,
   assertion (7), should already have refused that. Do NOT loosen `defaultMode` to 0444. (This is
   what CrashLooped every replica on 2026-11-08.)
2. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
   error about the tunnel `REPLACE-ME-TUNNEL-UUID` (and the origin-cert red herring). Do steps 1-5.
3. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
   credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
4. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
   from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed
   it.
5. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
   ```
   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
        hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
   ```
   (`ds/cilium` picks one node's agent; repeat per node if empty.)
6. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
   (step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
7. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
   node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.

## When to escalate or stop

- **Stop** if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not
  hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently
  overridden by `config.yaml`. Use the emergency stop under *Rollback*, then fix in git.
- **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
  **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
  a record created behind its back either fights it or is deleted by it, and a real hostname
  routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns
  (Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
- Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
  Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.

---

## Enrolling the tunnel (once)

cloudflared is **not** in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image
the Deployment uses, under podman — no host install. The working directory is
`~/.config/sops/cloudflared`: it is private to you, and it is the one host directory `dev.sh`
already mounts into the toolbox (as `/root/.config/sops`), so the toolbox's `sops` can read the
credentials JSON without it ever entering the repository tree.

```
CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba'
CF_DIR="$HOME/.config/sops/cloudflared"
mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR"
cf() { podman run --rm -it --user 0 --security-opt label=disable \
         -e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; }
```

(Keep `CF_IMG` in step with `platform/cloudflared/deployment.yaml` if Renovate has bumped it.)

### Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard)

Universal SSL covers only `keithjackson.dev` and `*.keithjackson.dev`; it does **not** cover the
second-level `*.home.keithjackson.dev`. In the `keithjackson.dev` zone:

1. **SSL/TLS → Edge Certificates → Advanced Certificate Manager**: subscribe.
2. Then EITHER **Order an advanced certificate** for exactly `home.keithjackson.dev,
   *.home.keithjackson.dev` (covers every name at once, independent of proxy state), OR turn on
   **Total TLS** (issues one certificate per *proxied* hostname as each one appears). The ordered
   certificate is the more predictable of the two for the later DNS flip; either satisfies the
   operator decision.
3. **Wait until the certificate shows `Active`** in the Edge Certificates list. Do not do step 6's
   canary before then — a TLS error at the edge would be the certificate, not the tunnel.
4. Leave **SSL/TLS encryption mode** at *Full (strict)*. (It does not govern tunnel origins —
   cloudflared verifies the Gateway itself — but it is the safe default for anything else.)

Edge settings that would break non-browser clients once traffic moves (check them now, in the zone):

- **Security → Bots → Bot Fight Mode: OFF.** It cannot be scoped by hostname or skipped by a WAF
  rule, and it challenges `git`, `docker`/registry pushes, Nextcloud sync clients and API calls.
- **Browser Integrity Check** and the **WAF / managed rules**: make sure neither challenges or
  blocks non-browser user agents on `gitea.home.keithjackson.dev` (git over HTTPS and the container
  registry under `/v2/`) or on API paths. If either does, add a WAF **Skip** custom rule for those
  hostnames/paths rather than turning protection off zone-wide.
- **Network → WebSockets: ON** (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC).

### Step 2 — log in and create the tunnel

```
cf tunnel login
# Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level
# origin certificate. Treat it as a credential.

cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s
# Prints "Created tunnel home-k3s with id <UUID>". The UUID is not secret; note it.
cf tunnel list
```

### Step 3 — put the UUID into the config

Do this in your own worktree on a new branch, as CLAUDE.md §0 requires:

```
UUID=<the id from step 2>
sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml
grep '^tunnel:' platform/cloudflared/config.yaml
```

### Step 4 — encrypt the credentials into the Secret, without printing them

Run from that worktree. `sops set` edits the already-encrypted document in place; the JSON goes
from the mounted file to `sops` inside the container and never reaches the terminal:

```
./infra/devcontainer/dev.sh run bash -c '
  set -euo pipefail
  cd /work
  f=/root/.config/sops/cloudflared/tunnel.json
  v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)"
  sops set platform/cloudflared/cloudflared-credentials.sops.yaml \
    "[\"stringData\"][\"credentials.json\"]" "$v"
  echo "credentials written"
'
```

(From a worktree, `dev.sh` needs `DEV_KUBECONFIG=/dev/null` and the `.git` mount shim — CLAUDE.md §3.
Or run this step from the main checkout and copy the encrypted file across.)

Verify — prints only `ok` or fails, never a value:

```
./infra/devcontainer/dev.sh run bash -c '
  cd /work
  UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml)
  sops -d --extract "[\"stringData\"][\"credentials.json\"]" \
      platform/cloudflared/cloudflared-credentials.sops.yaml \
    | jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \
      >/dev/null && echo ok'
```

Then store `tunnel.json` **and** `cert.pem` in your password manager, and delete them from disk:

```
rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem"
```

(`cert.pem` is only needed again to inspect (`tunnel info`), create or delete a tunnel. DNS records are
external-dns's, never `tunnel route dns`'s.)

### Step 5 — PR it

Commit `config.yaml` and the re-encrypted `cloudflared-credentials.sops.yaml`, gate, open a PR. No
other wiring is needed: the resource line and both overlay decryption patches already exist. On
merge Flux rolls the Deployment (the ConfigMap is content-hashed).

That PR touches `**/*.sops.yaml`, a sensitive path in `infra/gitea/approval-policy.json`, so
`approval-policy` will block it until you post `/approve-override <reason>` as a PR **review** of
type Comment (CLAUDE.md §1).

### Step 6 — verify

```
# Both replicas Ready, on different nodes:
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get pods -o wide'

# Logs: "Registered tunnel connection connIndex=0..3" on each replica.
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40'
```

From the workstation: `cf tunnel info home-k3s` (needs `cert.pem` back in `$CF_DIR` temporarily)
lists **two connectors**. In Grafana/Prometheus, `sum(cloudflared_tunnel_ha_connections)` is ~8 and
both alerts are clear.

**Canary — end to end, without touching any real record.** Hand-make ONE proxied CNAME for a
throwaway name (Cloudflare dashboard → DNS → Records → Add record):

    Type CNAME   Name tunnel-canary.home   Target <UUID>.cfargotunnel.com   Proxy status: Proxied

Do NOT use `cloudflared tunnel route dns` for it (see *When to escalate or stop*). The name hits the
wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it:

```
curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/
# expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge
# certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared
# could not reach or verify the Gateway; check the logs and `hubble observe` above.
```

**Then prove a REAL backend answers the tunnel's identity without a 403.** The canary cannot show
this: Envoy 404s an unknown host before it resolves any backend, and the `403 Access denied`
failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only
happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's
network identity, so it takes exactly the tunnel's path to the Gateway Service:

```
infra/devcontainer/dev.sh run bash -c '
  POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \
          -o jsonpath="{.items[0].metadata.name}")
  kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \
    --target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do
      printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \
        --connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"'
# expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress
# policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404.
# A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working.
```

(`kubectl debug` adds an ephemeral container: a cluster mutation, so an operator action. It
disappears with the pod on the next rollout.)

Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone:
`dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing.

## Rotating the tunnel credential

Do this whenever the credentials JSON (`TunnelSecret`) or `cert.pem` may have been exposed — a
transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this
workflow, so rotation is **a new tunnel**: the old credential stays valid until the old tunnel
is deleted, which is why the last step is not optional.

1. **Create the new tunnel** (*Enrolling*, step 2's `cf` helper; `cf tunnel login` again if
   `cert.pem` is not to hand). Use a NEW name — the old one is still taken:
   ```
   cf tunnel list                       # note the OLD tunnel's id and name
   cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2
   ```
2. **Swap the UUID** in your own worktree, on a new branch (CLAUDE.md §0):
   ```
   OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
   sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
   # Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
   # the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
   sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
   ```
3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
   same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
   `sops -d` to the terminal.
4. **Merge** — gate, PR, `/approve-override` by the operator (it touches `**/*.sops.yaml`), merge.
   Flux rolls the Deployment because the content-hashed ConfigMap changed.
5. **Verify** with *Enrolling* step 6: both replicas Ready, `Registered tunnel connection` in
   each replica's log, `cf tunnel info home-k3s-2` lists two connectors, and
   `cf tunnel info <old name>` lists **none**.
6. **Only then delete the old tunnel**, which revokes the exposed credential:
   ```
   cf tunnel cleanup <old name>   # drops any stale connector records first
   cf tunnel delete <old name>
   cf tunnel list                 # the old tunnel is gone
   ```
   Since Phase 2, records point at `<UUID>.cfargotunnel.com`: confirm external-dns has repointed
   every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those
   hostnames go dark.
   Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.

## Rollback

- **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and
  everything in it. No user traffic is affected either way. Optionally `cf tunnel delete home-k3s`
  (needs `cert.pem`) to retire the credential.
- **Emergency stop (operator, cluster mutation):** stop the connectors without waiting for a PR:
  ```
  infra/devcontainer/dev.sh run bash -c \
    'flux --context k3s-prod suspend kustomization cloudflared &&
     kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
  ```
  Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the
  names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then
  `flux resume kustomization cloudflared`.
- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
  (Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
  rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
  are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard**
  (`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the
  `pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not
  listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand:
  external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
  in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.