fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret

The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's
credentials.json down root:root 0440 — unreadable to the container. cloudflared
logged "permission denied" and then the red-herring "Error locating origin cert".

- deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440)
- L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the
  container's run-as uid/groups; mutation self-tested
- runbook: permission-denied troubleshooting entry, credential rotation section
- decisions/2026-11-08-cloudflared-creds-permission.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>

runbooks/cloudflare-tunnel.md

@@ -63,23 +63,36 @@ patch is missing; see causes below.
 
 ## Likely causes, most likely first
 
-1. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
-   error about the tunnel `REPLACE-ME-TUNNEL-UUID` or a missing origin certificate. Do steps 1-5.
-2. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
+**Read the FIRST error line, not the last.** When cloudflared cannot use the credentials file it
+falls back to looking for an origin certificate, and the error it ends on —
+`Cannot determine default origin certificate path` / `error parsing tunnel ID: Error locating
+origin cert` — is a **red herring** whatever the real cause was. The credentials file is the
+whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
+
+1. **`permission denied` on `credentials.json` → the "origin cert" error is a red herring.** Logs:
+   `couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission
+   denied`, then the origin-cert lines. The container (uid 65532) cannot read the Secret file:
+   the pod `securityContext.fsGroup: 65532` is missing, so the 0440 file is `root:root`. Fix it in
+   git (`platform/cloudflared/deployment.yaml`); the L0 check `cloudflared-ingress-rules`,
+   assertion (7), should already have refused that. Do NOT loosen `defaultMode` to 0444. (This is
+   what CrashLooped every replica on 2026-11-08.)
+2. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
+   error about the tunnel `REPLACE-ME-TUNNEL-UUID` (and the origin-cert red herring). Do steps 1-5.
+3. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
    credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
-3. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
+4. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
    from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed
    it.
-4. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
+5. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
    ```
    infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
         hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
    ```
    (`ds/cilium` picks one node's agent; repeat per node if empty.)
-5. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
+6. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
    (step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
-6. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
+7. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
    node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.
 
 ## When to escalate or stop
@@ -270,6 +283,42 @@ disappears with the pod on the next rollout.)
 Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone:
 `dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing.
 
+## Rotating the tunnel credential
+
+Do this whenever the credentials JSON (`TunnelSecret`) or `cert.pem` may have been exposed — a
+transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this
+workflow, so rotation is **a new tunnel**: the old credential stays valid until the old tunnel
+is deleted, which is why the last step is not optional.
+
+1. **Create the new tunnel** (*Enrolling*, step 2's `cf` helper; `cf tunnel login` again if
+   `cert.pem` is not to hand). Use a NEW name — the old one is still taken:
+   ```
+   cf tunnel list                       # note the OLD tunnel's id and name
+   cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2
+   ```
+2. **Swap the UUID** in your own worktree, on a new branch (CLAUDE.md §0):
+   ```
+   OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
+   sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
+   ```
+3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
+   same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
+   `sops -d` to the terminal.
+4. **Merge** — gate, PR, `/approve-override` by the operator (it touches `**/*.sops.yaml`), merge.
+   Flux rolls the Deployment because the content-hashed ConfigMap changed.
+5. **Verify** with *Enrolling* step 6: both replicas Ready, `Registered tunnel connection` in
+   each replica's log, `cf tunnel info home-k3s-2` lists two connectors, and
+   `cf tunnel info <old name>` lists **none**.
+6. **Only then delete the old tunnel**, which revokes the exposed credential:
+   ```
+   cf tunnel cleanup <old name>   # drops any stale connector records first
+   cf tunnel delete <old name>
+   cf tunnel list                 # the old tunnel is gone
+   ```
+   After the Phase 2 DNS flip, records point at `<UUID>.cfargotunnel.com`: confirm external-dns
+   has repointed every record to the NEW UUID before deleting, or those hostnames go dark.
+   Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
+
 ## Rollback
 
 - **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and