---
date: 2026-10-17
branch: fix/oauth2-proxy-oidc-startup-race
title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
---

## What changed

Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
serving yet, so the shared Gateway answers 503 and the proxy dies:

```
[provider.go:55] Performing OIDC Discovery...
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
```

It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
is already dead), and CrashLoopBackOff backoff is not tunable per-pod.

`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
accrue CrashLoopBackOff backoff and retries promptly, which is the point.

**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
for a pod wedged in `Init` forever, which is not an improvement.

## Why this shape and not the alternatives

* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
* **No new base image.** The chart already runs an `alpine` init container for its built-in
  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
  the chart already has a hook for.
* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
  the operator out of all of them, so the reversible, additive option wins.

The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.

## What I verified

* **The failure is measured, not theorised.** The log above is from the live pod
  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
  with `--previous` after the cold boot. The surviving replica's log shows the same startup
  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
  fault.
* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
  clean and both replicas have been healthy since.
* **The chart key is real**, at the pinned version range: `helm search repo` resolves
  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
  assumed from memory.
* The edited HelmRelease parses as YAML.

## What I did NOT verify

* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
  seven apps, so **do it when you can watch it**, not unattended.
* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
  from the measured failure, not demonstrated against it.
* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
  init container's log on first apply**: it should print "Keycloak discovery reachable" within
  seconds on a warm cluster, not the give-up message.
* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
  this should hold — unverified.
* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
  not a measured bound.
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
  subset only.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).