fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy #70
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Home-Network/home-network-k3s-impl#70
Reference in new issue
Block a user
The bug
Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC discovery once, at startup, and
exit 1s if it fails. On a cold boot Keycloak is not serving yet, so the shared Gateway answers 503 and the proxy dies:It then sits in exponential CrashLoopBackOff long after Keycloak is healthy — one of the two replicas was still down minutes later, and only a manual
kubectl delete podcleared it.There is no retry flag; the process does not survive a cold start. A readinessProbe cannot help — the container is already dead — and CrashLoopBackOff backoff is not tunable per-pod.
The fix
An
extraInitContainersgate (wait-for-idp) that polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod inInitaccrues no CrashLoopBackOff backoff and retries promptly, which is the whole point.Bounded, and fails open. After ~5 minutes it gives up and starts the proxy anyway, so the worst case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop for a pod wedged in
Initforever, which is not an improvement.Why this shape
extraInitContainersis the chart's own hook for exactly this — its commented example invalues.yamlis literally- name: wait-for-idp. The key was confirmed against the pinned chart (helm show values oauth2-proxy/oauth2-proxy --version 10.7.1), not assumed: a wrong Helm values key is silently ignored, which is the "reviewed, merged, apparently done, still broken" shape this repo keeps hitting.alpineinit container for its built-ininitContainers.waitForRedisgate. Pinned to3.22(verified to exist upstream) rather than the chart's ownlatest.postRenderers: works independently of the chart schema, but nothing in this tree uses postRenderers, so it would add a mechanism with no precedent for a problem the chart already has a hook for.--skip-oidc-discovery: removes the startup fetch entirely and is arguably more elegant, but it means hardcoding four Keycloak endpoints, and I could not verify offline that oauth2-proxy then fetches JWKS lazily rather than at startup. This component gates SSO for PiHole, the *arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks the operator out of all of them, so the additive, reversible option wins.Shell variables are written bare (
$i,$url,$(( ))) with no braced form:platformdeclarespostBuild.substituteFrom, and envsubst would blank a braced shell variable and silently break the loop. Same hazard and convention as the gitea/nextcloud OIDC bootstrap hooks;ci: flux-postbuild-substitutionenforces it.Verification
The failure is measured, not theorised — the log above is from the live pod (
lastState.terminated: {reason: Error, exitCode: 1}) captured with--previous, and the surviving replica's log shows the same startup succeeding once Keycloak was up, which is what makes it a race rather than a config fault.Not verified, and one point needs your eyes: this is unapplied — the init container has never run, and I did not re-create the race to prove the fix (that needs Keycloak down while oauth2-proxy starts; the honest test is the next cold boot). busybox
wget's TLS behaviour here is assumed: if it cannot validate the Gateway cert, the gate degrades safely (burns the 5-minute budget, then starts the proxy anyway) but the fast path would be silently dead. Check the init container's log on first apply — it should print "Keycloak discovery reachable" within seconds on a warm cluster, not the give-up message. Applying this rolls both replicas of the component gating SSO for seven apps, so do it when you can watch.Local gate: rc 3, zero failures — only
gitea-settings(CI-only token) and L2/L3conformance(CLUSTER_AVAILABLEunset) skipped, both structurally impossible locally per CLAUDE.md §3.Full detail:
decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md🤖 Generated with Claude Code
Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is not serving yet, the shared Gateway answers 503, and the proxy dies: [provider.go:55] Performing OIDC Discovery... [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC configuration: unexpected status "503": no healthy upstream It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was still down minutes later and only a manual pod delete cleared it. There is no retry flag, a readinessProbe cannot help (the container is already dead), and CrashLoopBackOff backoff is not tunable per-pod. Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the worst case is exactly today's behaviour and never something new. extraInitContainers is the chart's OWN hook for this -- its commented example is literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values key is silently ignored. No new base image: the chart already runs an alpine init container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream) rather than the chart's `latest`. Rejected: Flux postRenderers (no precedent in this tree, and the chart has a hook); --skip-oidc-discovery (removes the fetch entirely, but needs four hardcoded endpoints and I could not verify lazy-JWKS offline -- this component gates SSO for seven apps, so the additive option wins). Shell vars are written BARE with no braced form: platform declares postBuild.substituteFrom, and envsubst would blank a braced shell variable and silently break the loop. Same convention as the gitea/nextcloud OIDC hooks. UNAPPLIED; the init container has never run, and the race was not re-created to prove the fix. Check the init container's log on first apply. See the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>