jakman85andClaude Opus 5.5
d9ab57f2d1
The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's credentials.json down root:root 0440 — unreadable to the container. cloudflared logged "permission denied" and then the red-herring "Error locating origin cert". - deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440) - L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the container's run-as uid/groups; mutation self-tested - runbook: permission-denied troubleshooting entry, credential rotation section - decisions/2026-11-08-cloudflared-creds-permission.md Co-Authored-By: Claude Opus 5.5 <[email protected]>
4.2 KiB
4.2 KiB
date, branch, title
| date | branch | title |
|---|---|---|
| 2026-11-08 | fix/cloudflared-creds-permission | cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0 |
What changed
platform/cloudflared/deployment.yaml: podsecurityContext.fsGroup: 65532. Root cause of the CrashLoop that followed #152/#153: the pod runs as65532:65532with nofsGroup, and the Secret volume isdefaultMode: 0440. The kubelet therefore laidcredentials.jsondown asroot:root 0440, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared loggedcouldn't read tunnel credentials ... permission denied, then fell back to origin-cert lookup and ended on the misleadingerror parsing tunnel ID: Error locating origin cert. WithfsGroupthe files are group-owned by 65532, readable by the container and no other uid; the mode stays 0440 (not loosened to 0444). Same fix asplatform/keycloak/client-secret-reconciler.yaml. Still non-root,runAsNonRoot: true, unchanged otherwise.conformance/tests/l0-static/cloudflared-ingress-rules/check.sh: new assertion (7). For each Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups (runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset runAsUser fails (image default is not knowable offline). Conservative: it does not credit the kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400, runAsUser unset must each be flagged. Chainsaw step description updated to match.runbooks/cloudflare-tunnel.md: new first "likely cause" (permission denied on credentials.json → the origin-cert error is a red herring) plus a "read the first error line" note; causes renumbered. New section Rotating the tunnel credential: create a new tunnel, swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old has none), thencloudflared tunnel cleanup/delete <old>.- NOT touched, on purpose:
tunnel:inconfig.yamlandcloudflared-credentials.sops.yaml. The operator is rotating the credential (it was exposed in a transcript) in a separate PR.
What I verified
- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}, nofsGroup; creds volumedefaultMode: 288(= 0440). All three pods CrashLoopBackOff (38 restarts). Secretcloudflared-credentialshas the one keycredentials.json(key names only). - Pinned image
cloudflare/cloudflared@sha256:4955524a...config (skopeo inspect --config):User: 65532:65532,Entrypoint: [cloudflared],Cmd: [version]. - The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are 0:0 mode 0440, and PASSES (51 assertions) with the fix. All three mutants are flagged. - Kyverno admission policies under
platform/kyverno/admission/that mention securityContext fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here. - Toolbox gate: see the PR description for the rc and skipped stages.
What I did NOT verify
- That the pods actually become Ready after merge. Not deployed by me; Flux applies it. The file mode inside the running container was not observed (pods crash before an exec is possible, and exec would be an operator action anyway). The readability claim rests on the kubelet's documented fsGroup behaviour and the identical, working keycloak reconciler.
- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot: the operator is rotating it.
- The rotation runbook section has not been executed;
cloudflared tunnel cleanup/deletesyntax is from the CLI's documented commands, not run here. - The deployment comment says the image ENTRYPOINT is
cloudflared --no-autoupdate; the pinned image's ENTRYPOINT is plaincloudflared. Left unchanged (out of scope); flagged for follow-up.