fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret #154

2026-11-08T21:54:42Z

gitea-admin commented

2026-11-08 21:54:42 +00:00

Problem

platform/cloudflared CrashLoops on k3s-prod (all replicas, 38+ restarts):
couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission denied, followed by the red herring error parsing tunnel ID: Error locating origin cert.

Root cause (measured)

Live pod securityContext: runAsUser: 65532, runAsGroup: 65532, no fsGroup. Creds Secret volume defaultMode: 288 (0440).
Pinned image config (skopeo inspect --config): User: 65532:65532.
Without fsGroup the kubelet writes the Secret files root:root 0440, which uid 65532 cannot read. The keycloak client-secret reconciler hit the same bug and fixed it the same way.

Fix

platform/cloudflared/deployment.yaml: pod fsGroup: 65532. The mode stays 0440 and the pod stays non-root. No Kyverno policy constrains fsGroup for this namespace.
L0 cloudflared-ingress-rules assertion (7): every Secret volume must be readable by the container's effective uid/groups (kubelet ownership model; unset runAsUser = fail). It is self-tested by three mutants. It fails (rc 1) on the pre-fix manifest with the exact cause, and passes (51 assertions) with the fix.
runbooks/cloudflare-tunnel.md: a troubleshooting entry for "permission denied → origin-cert error is a red herring", and a new Rotating the tunnel credential section.
decisions/2026-11-08-cloudflared-creds-permission.md.

This PR does not touch tunnel: or cloudflared-credentials.sops.yaml. Credential rotation is the operator's separate PR.

Gate (pinned toolbox, from a worktree, committed tree)

./ci/gate.sh → rc 3. The only stage that did not run is gitea-settings (no SETTINGS_API_TOKEN locally). spec-immutability: pass (spec repo mounted). L2/L3 conformance: scaffold-skip (no cluster), as usual. Local approval-policy reports N/A off-PR.

Sensitive paths

Touches conformance/** (conformance/tests/l0-static/cloudflared-ingress-rules/{check.sh,chainsaw-test.yaml}), so approval-policy will block until the operator posts /approve-override <reason> as a PR review of type Comment. If the fix is wanted faster, the deployment.yaml change alone is not sensitive and could go in a separate PR.

Unverified

That the pods go Ready after Flux applies this. The in-container file mode was not observed. The validity of the current credential is moot because it is being rotated. The rotation steps have not been executed.

🤖 Generated with Claude Code

## Problem platform/cloudflared CrashLoops on k3s-prod (all replicas, 38+ restarts): `couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission denied`, followed by the red herring `error parsing tunnel ID: Error locating origin cert`. ## Root cause (measured) - Live pod securityContext: `runAsUser: 65532, runAsGroup: 65532`, **no `fsGroup`**. Creds Secret volume `defaultMode: 288` (0440). - Pinned image config (`skopeo inspect --config`): `User: 65532:65532`. - Without fsGroup the kubelet writes the Secret files `root:root 0440`, which uid 65532 cannot read. The keycloak client-secret reconciler hit the same bug and fixed it the same way. ## Fix - `platform/cloudflared/deployment.yaml`: pod `fsGroup: 65532`. The mode stays 0440 and the pod stays non-root. No Kyverno policy constrains fsGroup for this namespace. - L0 `cloudflared-ingress-rules` assertion (7): every Secret volume must be readable by the container's effective uid/groups (kubelet ownership model; unset runAsUser = fail). It is self-tested by three mutants. **It fails (rc 1) on the pre-fix manifest** with the exact cause, and passes (51 assertions) with the fix. - `runbooks/cloudflare-tunnel.md`: a troubleshooting entry for "permission denied → origin-cert error is a red herring", and a new **Rotating the tunnel credential** section. - `decisions/2026-11-08-cloudflared-creds-permission.md`. This PR does **not** touch `tunnel:` or `cloudflared-credentials.sops.yaml`. Credential rotation is the operator's separate PR. ## Gate (pinned toolbox, from a worktree, committed tree) `./ci/gate.sh` → **rc 3**. The only stage that did not run is `gitea-settings` (no SETTINGS_API_TOKEN locally). `spec-immutability`: pass (spec repo mounted). L2/L3 conformance: scaffold-skip (no cluster), as usual. Local `approval-policy` reports N/A off-PR. ## Sensitive paths Touches `conformance/**` (`conformance/tests/l0-static/cloudflared-ingress-rules/{check.sh,chainsaw-test.yaml}`), so `approval-policy` will block until **the operator** posts `/approve-override <reason>` as a PR review of type Comment. If the fix is wanted faster, the `deployment.yaml` change alone is not sensitive and could go in a separate PR. ## Unverified That the pods go Ready after Flux applies this. The in-container file mode was not observed. The validity of the current credential is moot because it is being rotated. The rotation steps have not been executed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-11-08 21:54:42 +00:00

fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret

gate / gate (pull_request) Successful in 42s

Details

gate / conformance (pull_request) Skipped

Details

d9ab57f2d1

The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's
credentials.json down root:root 0440 — unreadable to the container. cloudflared
logged "permission denied" and then the red-herring "Error locating origin cert".

- deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440)
- L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the
  container's run-as uid/groups; mutation self-tested
- runbook: permission-denied troubleshooting entry, credential rotation section
- decisions/2026-11-08-cloudflared-creds-permission.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jakman85 approved these changes

2026-11-08 21:56:53 +00:00

jakman85 scheduled this pull request to auto merge when all checks succeed 2026-11-08 21:57:09 +00:00

jakman85 merged commit 16222606ae into main

2026-11-08 21:57:47 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#154