fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret

The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's
credentials.json down root:root 0440 — unreadable to the container. cloudflared
logged "permission denied" and then the red-herring "Error locating origin cert".

- deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440)
- L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the
  container's run-as uid/groups; mutation self-tested
- runbook: permission-denied troubleshooting entry, credential rotation section
- decisions/2026-11-08-cloudflared-creds-permission.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>

decisions/2026-11-08-cloudflared-creds-permission.md

@@ -0,0 +1,59 @@
+---
+date: 2026-11-08
+branch: fix/cloudflared-creds-permission
+title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
+---
+
+## What changed
+
+- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
+  the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
+  Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
+  `root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
+  `couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
+  and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
+  `fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
+  mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
+  Still non-root, `runAsNonRoot: true`, unchanged otherwise.
+- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
+  Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
+  mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
+  (runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
+  runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
+  kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
+  runAsUser unset must each be flagged. Chainsaw step description updated to match.
+- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
+  credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
+  note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
+  swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
+  has none), then `cloudflared tunnel cleanup`/`delete <old>`.
+- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
+  operator is rotating the credential (it was exposed in a transcript) in a separate PR.
+
+## What I verified
+
+- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
+  `{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
+  `fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
+  restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
+- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
+  `User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
+- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
+  `container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
+  0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
+- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
+  fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
+- Toolbox gate: see the PR description for the rc and skipped stages.
+
+## What I did NOT verify
+
+- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
+  file mode inside the running container was not observed (pods crash before an exec is possible,
+  and exec would be an operator action anyway). The readability claim rests on the kubelet's
+  documented fsGroup behaviour and the identical, working keycloak reconciler.
+- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
+  the operator is rotating it.
+- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
+  syntax is from the CLI's documented commands, not run here.
+- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
+  image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.