home-network-k3s-impl/decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

date, branch, title

date	branch	title
2026-10-17	fix/oauth2-proxy-oidc-startup-race	oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves

What changed

Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC discovery once, at startup, and os.Exit(1)s if it fails. On a cold boot Keycloak is not serving yet, so the shared Gateway answers 503 and the proxy dies:

[provider.go:55] Performing OIDC Discovery...
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
             failed to discover OIDC configuration: unexpected status "503": no healthy upstream

It then sits in exponential CrashLoopBackOff long after Keycloak is healthy — one of the two replicas was still down minutes later and only a manual kubectl delete pod cleared it. There is no retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container is already dead), and CrashLoopBackOff backoff is not tunable per-pod.

platform/oauth2-proxy/helmrelease.yaml gains an extraInitContainers gate (wait-for-idp) that polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the point.

Bounded, and fails open. After ~5 minutes it gives up and starts the proxy anyway, so the worst case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop for a pod wedged in Init forever, which is not an improvement.

Why this shape and not the alternatives

• extraInitContainers is the chart's own documented hook for precisely this. Its commented example in values.yaml is literally - name: wait-for-idp. The key was confirmed against the pinned chart (helm show values oauth2-proxy/oauth2-proxy --version 10.7.1 → extraInitContainers: [] at line 428), not assumed — a wrong Helm values key is silently ignored, which is the "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
• No new base image. The chart already runs an alpine init container for its built-in initContainers.waitForRedis gate, so alpine is a base this chart uses regardless. Pinned to 3.22 (verified to exist upstream) rather than the chart's own latest.
• Flux postRenderers rejected: it would work independently of the chart schema, but nothing in this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem the chart already has a hook for.
• --skip-oidc-discovery rejected: it removes the startup fetch entirely, which is arguably more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole, the *arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks the operator out of all of them, so the reversible, additive option wins.

The $$-free convention matters here: shell variables are written bare ($i, $url, $(( ))) with no braced form anywhere, because platform declares postBuild.substituteFrom cluster-vars and envsubst consumes braced expansions — a braced shell variable renders to the empty string and silently breaks the loop. ${BASE_DOMAIN} is braced on purpose. Same hazard and same convention as the gitea and nextcloud OIDC bootstrap hooks, and ci: flux-postbuild-substitution enforces it.

What I verified

• The failure is measured, not theorised. The log above is from the live pod (oauth2-proxy-5d6c74876b-bzb2p, lastState.terminated: {reason: Error, exitCode: 1}), captured with --previous after the cold boot. The surviving replica's log shows the same startup succeeding once Keycloak was up, which is what identifies this as a race rather than a config fault.
• Deleting the pod is a sufficient workaround — done on prod 2026-10-17, the replacement came up clean and both replicas have been healthy since.
• The chart key is real, at the pinned version range: helm search repo resolves >=10.0.0 <11.0.0 to chart 10.7.1 (app 7.15.5), whose values carry extraInitContainers: [], extraContainers: [] and the built-in initContainers.waitForRedis (which uses alpine).
• docker.io/library/alpine:3.22 exists — confirmed against the registry tag list rather than assumed from memory.
• The edited HelmRelease parses as YAML.

What I did NOT verify

• UNAPPLIED. The init container has never run. I did not reconcile the HelmRelease, so I have not seen wait-for-idp execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for seven apps, so do it when you can watch it, not unattended.
• I did not re-create the cold-start race to prove the fix. That needs Keycloak down while oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned from the measured failure, not demonstrated against it.
• busybox wget's TLS behaviour here is assumed, not tested. The claim is that alpine's ca-certificates-bundle lets it validate the Gateway's LE-prod certificate, so a genuine TLS failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy anyway — but the fast path would be silently dead and this would be doing nothing. Check the init container's log on first apply: it should print "Keycloak discovery reachable" within seconds on a warm cluster, not the give-up message.
• Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not re-proven. The main container does exactly this fetch today and succeeds, so the path is known good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname (decisions/2026-10-04-plex-advertise-plexdirect.md); oauth2-proxy is exempt only because Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so this should hold — unverified.
• 60 attempts x 5s is not tuned. It is "comfortably longer than Keycloak's observed cold start", not a measured bound.
• No runtime conformance. vars.CLUSTER_AVAILABLE is unset, so CI proves the L0/L1 offline subset only.
• I did not re-measure branch protection or runner state (CLAUDE.md §2b).