jakman85andClaude Opus 5.5
d9ab57f2d1
The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's credentials.json down root:root 0440 — unreadable to the container. cloudflared logged "permission denied" and then the red-herring "Error locating origin cert". - deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440) - L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the container's run-as uid/groups; mutation self-tested - runbook: permission-denied troubleshooting entry, credential rotation section - decisions/2026-11-08-cloudflared-creds-permission.md Co-Authored-By: Claude Opus 5.5 <[email protected]>
60 lines
4.2 KiB
Markdown
60 lines
4.2 KiB
Markdown
---
|
|
date: 2026-11-08
|
|
branch: fix/cloudflared-creds-permission
|
|
title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
|
|
---
|
|
|
|
## What changed
|
|
|
|
- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
|
|
the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
|
|
Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
|
|
`root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
|
|
`couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
|
|
and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
|
|
`fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
|
|
mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
|
|
Still non-root, `runAsNonRoot: true`, unchanged otherwise.
|
|
- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
|
|
Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
|
|
mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
|
|
(runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
|
|
runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
|
|
kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
|
|
runAsUser unset must each be flagged. Chainsaw step description updated to match.
|
|
- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
|
|
credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
|
|
note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
|
|
swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
|
|
has none), then `cloudflared tunnel cleanup`/`delete <old>`.
|
|
- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
|
|
operator is rotating the credential (it was exposed in a transcript) in a separate PR.
|
|
|
|
## What I verified
|
|
|
|
- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
|
|
`{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
|
|
`fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
|
|
restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
|
|
- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
|
|
`User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
|
|
- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
|
|
`container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
|
|
0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
|
|
- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
|
|
fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
|
|
- Toolbox gate: see the PR description for the rc and skipped stages.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
|
|
file mode inside the running container was not observed (pods crash before an exec is possible,
|
|
and exec would be an operator action anyway). The readability claim rests on the kubelet's
|
|
documented fsGroup behaviour and the identical, working keycloak reconciler.
|
|
- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
|
|
the operator is rotating it.
|
|
- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
|
|
syntax is from the CLI's documented commands, not run here.
|
|
- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
|
|
image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.
|