---
date: 2026-11-08
branch: fix/cloudflared-creds-permission
title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
---

## What changed

- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
  the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
  Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
  `root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
  `couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
  and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
  `fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
  mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
  Still non-root, `runAsNonRoot: true`, unchanged otherwise.
- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
  Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
  mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
  (runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
  runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
  kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
  runAsUser unset must each be flagged. Chainsaw step description updated to match.
- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
  credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
  note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
  swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
  has none), then `cloudflared tunnel cleanup`/`delete <old>`.
- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
  operator is rotating the credential (it was exposed in a transcript) in a separate PR.

## What I verified

- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
  `{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
  `fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
  restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
  `User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
  `container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
  0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
  fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
- Toolbox gate: see the PR description for the rc and skipped stages.

## What I did NOT verify

- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
  file mode inside the running container was not observed (pods crash before an exec is possible,
  and exec would be an operator action anyway). The readability claim rests on the kubelet's
  documented fsGroup behaviour and the identical, working keycloak reconciler.
- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
  the operator is rotating it.
- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
  syntax is from the CLI's documented commands, not run here.
- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
  image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.