jakman85andClaude Opus 5
c150327a30
Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC
discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is
not serving yet, the shared Gateway answers 503, and the proxy dies:
[provider.go:55] Performing OIDC Discovery...
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover
OIDC configuration: unexpected status "503": no healthy upstream
It then sits in exponential CrashLoopBackOff long after Keycloak is healthy --
one replica was still down minutes later and only a manual pod delete cleared
it. There is no retry flag, a readinessProbe cannot help (the container is
already dead), and CrashLoopBackOff backoff is not tunable per-pod.
Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc
until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries
promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the
worst case is exactly today's behaviour and never something new.
extraInitContainers is the chart's OWN hook for this -- its commented example is
literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned
chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values
key is silently ignored. No new base image: the chart already runs an alpine init
container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream)
rather than the chart's `latest`.
Rejected: Flux postRenderers (no precedent in this tree, and the chart has a
hook); --skip-oidc-discovery (removes the fetch entirely, but needs four
hardcoded endpoints and I could not verify lazy-JWKS offline -- this component
gates SSO for seven apps, so the additive option wins).
Shell vars are written BARE with no braced form: platform declares
postBuild.substituteFrom, and envsubst would blank a braced shell variable and
silently break the loop. Same convention as the gitea/nextcloud OIDC hooks.
UNAPPLIED; the init container has never run, and the race was not re-created to
prove the fix. Check the init container's log on first apply. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
144 lines
7.8 KiB
YAML
144 lines
7.8 KiB
YAML
---
|
|
# oauth2-proxy — forward-auth for apps WITHOUT usable OIDC (PiHole, *arr UIs, qBittorrent, Homepage,
|
|
# Prometheus, Alertmanager, Hubble UI — ADR-0014). Authenticates users against the Keycloak `homelab`
|
|
# realm, then the Gateway ext-auth lets the request through only for an authenticated session
|
|
# (R-IDP-02: no management UI unauthenticated). Chart image is quay.io/oauth2-proxy (not Bitnami).
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: oauth2-proxy
|
|
namespace: oauth2-proxy
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: oauth2-proxy
|
|
version: ">=10.0.0 <11.0.0" # k8s-1.36 catch-up (app v7.15.x). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: oauth2-proxy
|
|
namespace: oauth2-proxy
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
replicaCount: ${OAUTH2_PROXY_REPLICAS} # overlay knob: dev 1 / prod 2 (HA) — R-ENV-02
|
|
config:
|
|
# client-id / client-secret / cookie-secret from the SOPS secret (name only). client-id +
|
|
# client-secret MUST match the realm's `oauth2-proxy` client (keycloak-realm-clients secret).
|
|
existingSecret: oauth2-proxy-credentials
|
|
extraArgs:
|
|
provider: oidc
|
|
oidc-issuer-url: https://keycloak.${BASE_DOMAIN}/realms/homelab
|
|
# Forward-auth (ext-auth) mode: no upstream app; return 202 for an authenticated session so the
|
|
# Gateway ext-authz can allow/deny. Auth headers are surfaced to the backend.
|
|
reverse-proxy: "true"
|
|
upstream: static://202
|
|
set-xauthrequest: "true"
|
|
pass-access-token: "true"
|
|
email-domain: "*"
|
|
# Single sign-in/callback host for all forward-auth-protected services on the platform zone.
|
|
redirect-url: https://auth.${BASE_DOMAIN}/oauth2/callback
|
|
whitelist-domain: ".${BASE_DOMAIN}"
|
|
cookie-domain: ".${BASE_DOMAIN}"
|
|
cookie-secure: "true"
|
|
# Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The
|
|
# Keycloak `oauth2-proxy` client emits a `groups` claim (full.path=false => bare names like
|
|
# `platform-admins`, keycloak-realm-import.yaml). `--allowed-groups` below is the PLATFORM FLOOR:
|
|
# a valid Keycloak session is NOT enough — the user must ALSO be in `platform-admins`, or
|
|
# /oauth2/auth returns 403 and ext_authz denies (R-IDP-06). Every currently-onboarded
|
|
# forward-auth app (pihole, *arr, qbittorrent, homepage, prometheus, alertmanager, hubble)
|
|
# declares `groups: [platform-admins]` in its ServiceDefinition, so this floor matches them all.
|
|
# Per-ROUTE group differentiation (a future app allowing a different group) does NOT need a
|
|
# second proxy: the ext_authz filter appends `?allowed_groups=<group>` to /oauth2/auth from the
|
|
# HTTPRoute's `homelab.keithjackson.dev/auth-groups` annotation — oauth2-proxy honours the
|
|
# per-request query on top of this floor. See forward-auth-cec.yaml + README "opt-in".
|
|
oidc-groups-claim: groups
|
|
allowed-group: platform-admins # R-IDP-06 floor: authenticated AND in platform-admins (oauth2-proxy flag is singular --allowed-group, repeatable)
|
|
# Single Keycloak provider => skip the oauth2-proxy interstitial "Sign In" button and 302
|
|
# straight to Keycloak. Verified in-cluster: WITHOUT this, an ext_authz-denied browser request
|
|
# gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
|
|
skip-provider-button: "true"
|
|
scope: "openid email profile"
|
|
# ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
|
|
# MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
|
|
# startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
|
|
# the Gateway answers 503 and the proxy dies:
|
|
#
|
|
# [provider.go:55] Performing OIDC Discovery...
|
|
# [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
|
|
# configuration: unexpected status "503": no healthy upstream
|
|
#
|
|
# It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
|
|
# still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
|
|
# process simply does not survive a cold start. A readinessProbe cannot help (the container is
|
|
# already dead), and CrashLoopBackOff backoff is not tunable per-pod.
|
|
#
|
|
# `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
|
|
# values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
|
|
# container for its built-in waitForRedis gate, so this adds no new base image to the platform.
|
|
# A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
|
|
# point.
|
|
#
|
|
# BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
|
|
# case is then exactly today's behaviour, never something new. An unbounded wait would trade a
|
|
# CrashLoop for a pod wedged in Init forever, which is not an improvement.
|
|
#
|
|
# SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
|
|
# rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
|
|
# CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
|
|
# silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
|
|
# knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
|
|
# OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
|
|
extraInitContainers:
|
|
- name: wait-for-idp
|
|
image: docker.io/library/alpine:3.22
|
|
imagePullPolicy: IfNotPresent
|
|
command: ["/bin/sh", "-c"]
|
|
args:
|
|
- |
|
|
set -u
|
|
url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
|
i=0
|
|
while [ "$i" -lt 60 ]; do
|
|
# busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
|
|
# failure is a non-zero exit here rather than a false pass.
|
|
if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
|
|
echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
|
|
exit 0
|
|
fi
|
|
i=$(( i + 1 ))
|
|
sleep 5
|
|
done
|
|
echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
|
|
echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
|
|
exit 0
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
runAsNonRoot: true
|
|
runAsUser: 65534
|
|
runAsGroup: 65534
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 16Mi
|
|
limits:
|
|
memory: 32Mi
|
|
|
|
# Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
|
|
metrics:
|
|
enabled: true
|
|
# ENFORCEMENT (Phase 6): oauth2-proxy is the AUTHENTICATOR; the per-route ext_authz ENFORCEMENT is
|
|
# Gateway-side. On Cilium 1.16.5 the shared Gateway's Envoy config is a machine-owned, constantly
|
|
# regenerated CiliumEnvoyConfig (`gateway/cilium-gateway-shared`) that CANNOT be injected into,
|
|
# and native Gateway-API ExternalAuth (GEP-1494) ships in NO released Cilium version yet. The
|
|
# enforcement mechanism + the three wiring options are documented in README.md; the interim
|
|
# self-authored ext_authz listener is forward-auth-cec.yaml (a template, not yet Flux-wired).
|