From c150327a30697b3fca234f78c2664f10127faa24 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 17 Oct 2026 22:47:30 -0500 Subject: [PATCH] fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is not serving yet, the shared Gateway answers 503, and the proxy dies: [provider.go:55] Performing OIDC Discovery... [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC configuration: unexpected status "503": no healthy upstream It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was still down minutes later and only a manual pod delete cleared it. There is no retry flag, a readinessProbe cannot help (the container is already dead), and CrashLoopBackOff backoff is not tunable per-pod. Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the worst case is exactly today's behaviour and never something new. extraInitContainers is the chart's OWN hook for this -- its commented example is literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values key is silently ignored. No new base image: the chart already runs an alpine init container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream) rather than the chart's `latest`. Rejected: Flux postRenderers (no precedent in this tree, and the chart has a hook); --skip-oidc-discovery (removes the fetch entirely, but needs four hardcoded endpoints and I could not verify lazy-JWKS offline -- this component gates SSO for seven apps, so the additive option wins). Shell vars are written BARE with no braced form: platform declares postBuild.substituteFrom, and envsubst would blank a braced shell variable and silently break the loop. Same convention as the gitea/nextcloud OIDC hooks. UNAPPLIED; the init container has never run, and the race was not re-created to prove the fix. Check the init container's log on first apply. See the handoff. Co-Authored-By: Claude Opus 5 (1M context) --- ...26-10-17-oauth2-proxy-oidc-startup-race.md | 99 +++++++++++++++++++ platform/oauth2-proxy/helmrelease.yaml | 69 +++++++++++++ 2 files changed, 168 insertions(+) create mode 100644 decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md diff --git a/decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md b/decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md new file mode 100644 index 0000000..ed0d588 --- /dev/null +++ b/decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md @@ -0,0 +1,99 @@ +--- +date: 2026-10-17 +branch: fix/oauth2-proxy-oidc-startup-race +title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves" +--- + +## What changed + +Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC +discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not +serving yet, so the shared Gateway answers 503 and the proxy dies: + +``` +[provider.go:55] Performing OIDC Discovery... +[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ... + failed to discover OIDC configuration: unexpected status "503": no healthy upstream +``` + +It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two +replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no +retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container +is already dead), and CrashLoopBackOff backoff is not tunable per-pod. + +`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that +polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not +accrue CrashLoopBackOff backoff and retries promptly, which is the point. + +**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst +case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop +for a pod wedged in `Init` forever, which is not an improvement. + +## Why this shape and not the alternatives + +* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented + example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the + pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []` + at line 428), not assumed — a wrong Helm values key is silently ignored, which is the + "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by. +* **No new base image.** The chart already runs an `alpine` init container for its built-in + `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to + `3.22` (verified to exist upstream) rather than the chart's own `latest`. +* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in + this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem + the chart already has a hook for. +* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably + more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that + oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole, + the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks + the operator out of all of them, so the reversible, additive option wins. + +The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`) +with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars` +and envsubst consumes braced expansions — a braced shell variable renders to the empty string and +silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as +the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it. + +## What I verified + +* **The failure is measured, not theorised.** The log above is from the live pod + (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured + with `--previous` after the cold boot. The surviving replica's log shows the same startup + succeeding once Keycloak was up, which is what identifies this as a race rather than a config + fault. +* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up + clean and both replicas have been healthy since. +* **The chart key is real**, at the pinned version range: `helm search repo` resolves + `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`, + `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`). +* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than + assumed from memory. +* The edited HelmRelease parses as YAML. + +## What I did NOT verify + +* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not + seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy + replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for + seven apps, so **do it when you can watch it**, not unattended. +* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while + oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned + from the measured failure, not demonstrated against it. +* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's + `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS + failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate + degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy + anyway — but the fast path would be silently dead and this would be doing nothing. **Check the + init container's log on first apply**: it should print "Keycloak discovery reachable" within + seconds on a warm cluster, not the give-up message. +* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not + re-proven.** The main container does exactly this fetch today and succeeds, so the path is known + good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname + (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because + Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so + this should hold — unverified. +* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start", + not a measured bound. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline + subset only. +* I did not re-measure branch protection or runner state (CLAUDE.md §2b). diff --git a/platform/oauth2-proxy/helmrelease.yaml b/platform/oauth2-proxy/helmrelease.yaml index 5e4ce98..7e41801 100644 --- a/platform/oauth2-proxy/helmrelease.yaml +++ b/platform/oauth2-proxy/helmrelease.yaml @@ -63,6 +63,75 @@ spec: # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX). skip-provider-button: "true" scope: "openid email profile" + # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ────────── + # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at + # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so + # the Gateway answers 503 and the proxy dies: + # + # [provider.go:55] Performing OIDC Discovery... + # [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC + # configuration: unexpected status "503": no healthy upstream + # + # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was + # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the + # process simply does not survive a cold start. A readinessProbe cannot help (the container is + # already dead), and CrashLoopBackOff backoff is not tunable per-pod. + # + # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in + # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init + # container for its built-in waitForRedis gate, so this adds no new base image to the platform. + # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole + # point. + # + # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst + # case is then exactly today's behaviour, never something new. An unbounded wait would trade a + # CrashLoop for a pod wedged in Init forever, which is not an improvement. + # + # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is + # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which + # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and + # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars + # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud + # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.) + extraInitContainers: + - name: wait-for-idp + image: docker.io/library/alpine:3.22 + imagePullPolicy: IfNotPresent + command: ["/bin/sh", "-c"] + args: + - | + set -u + url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration + i=0 + while [ "$i" -lt 60 ]; do + # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS + # failure is a non-zero exit here rather than a false pass. + if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then + echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy" + exit 0 + fi + i=$(( i + 1 )) + sleep 5 + done + echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2 + echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2 + exit 0 + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + memory: 32Mi + # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config). metrics: enabled: true