fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment

MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster
DNS was a single pod on a single node, so draining or rebooting that node was a
CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service
lookup, every CNPG replica finding its primary, every operator addressing the
API by name. It was invisible because 1/1 Ready looks healthy.

It is urgent rather than latent now: the scheduled reboot automation on
feat/automated-upgrades drains and reboots one node at a time, unattended,
overnight -- turning a single point of failure into a recurring one.

WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns
Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml,
managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash
label. k3s re-stages its bundled manifests on startup, so replicas: 2 written
there -- or a kubectl scale -- is reverted on the next restart or upgrade.

Rejected alternatives:
* coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern).
  Deterministic, but takes ownership of coredns from k3s permanently so k3s
  upgrades stop updating it. Too high a price for a replica count.
* A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy
  controller would both own one object and fight -- the default-deny
  NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit.

An HPA is a separate object k3s does not own, driving /scale rather than the
Deployment spec, so there is no shared-ownership fight; if k3s re-stages
replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief
window at 1 replica after a k3s restart.

It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the
autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%),
so the 80% target is never approached. scaleDown stabilization 600s because DNS
is the worst thing to flap; scaleUp immediate, since that restores redundancy.

The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds
the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched.

UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on
restart is reasoned, not observed. After merge check `describe hpa coredns` for
ScalingActive=True -- an inert HPA looks identical to a working one because the
hand scale already left 2 replicas. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1

catalog/plex.yaml

@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-18-coredns-ha.md

@@ -0,0 +1,108 @@
+---
+date: 2026-10-18
+branch: fix/coredns-ha
+title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
+---
+
+## What changed
+
+**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
+single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
+outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
+finding its primary, every operator addressing the API by name. It was invisible because `1/1
+Ready` looks perfectly healthy.
+
+It also became urgent rather than latent: the scheduled reboot automation on
+`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
+single point of failure into a recurring one.
+
+New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
+`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
+alongside the split-horizon ConfigMap already there.
+
+**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
+and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
+(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
+`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
+bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
+on the next k3s restart or upgrade. Two alternatives were weighed and rejected:
+
+* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
+  file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
+  from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
+  by hand forever. Too high a price for a replica count.
+* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
+  would both own one object and fight continuously — the exact shape of the `default-deny`
+  NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).
+
+An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
+the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
+sees it below `minReplicas` and scales back within one reconcile.
+
+It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
+HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
+thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
+after a k3s restart.
+
+**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
+--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
+rather than after merge. That hand scale is what this PR makes durable.
+
+## What I verified
+
+* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
+* **k3s ownership:** the `AddOn/coredns` object exists with source
+  `/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
+  `objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
+  k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
+  `platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
+  Deployment or its replica count.
+* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
+  in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
+  10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
+  checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
+  re-applies on manifest change and on restart.
+* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
+  Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
+  verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
+  request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
+  sit at `minReplicas`.
+* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
+  and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
+  there.
+* All three touched/added YAML files parse.
+
+## What I did NOT verify
+
+* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
+  reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
+  Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
+  `ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
+  which looks identical to working because the hand scale already left 2 replicas in place. **Check
+  `kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
+* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
+  on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
+  **not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
+  operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
+  (though harmless).
+* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
+  write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
+  at 1 replica. Neither the window's length nor the outcome has been measured.
+* **I nearly shipped a destructive change.** My first write of
+  `platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
+  would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
+  DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
+  the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
+  already existed and I assumed it did not. **No conformance check would obviously have caught the
+  removal** — it is valid YAML and a legitimate-looking prune.
+* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
+  replicas could in principle land on one node and a single drain would still take DNS down. I did
+  not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
+  neither is in this change.
+* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
+  changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
+* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
+  noticing.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

platform/coredns/hpa.yaml

@@ -0,0 +1,112 @@
+---
+# =================================================================================================
+# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
+# =================================================================================================
+# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
+# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
+# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
+# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
+# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
+# because `1/1 Ready` looks perfectly healthy.
+#
+# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
+# reboots one node at a time on a schedule, which turns a latent single point of failure into a
+# recurring one, unattended, in the middle of the night.
+#
+# -------------------------------------------------------------------------------------------------
+# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
+# -------------------------------------------------------------------------------------------------
+# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
+# deploy controller from a bundled manifest, not by Flux:
+#
+#   $ kubectl -n kube-system get addon coredns
+#   NAME      SOURCE                                        CHECKSUM
+#   coredns   /data/k3s/server/manifests/coredns.yaml        8cd91595...
+#   $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
+#   deploy@k3s-prod-1 k3s
+#
+# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
+# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
+# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
+# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
+# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
+# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
+#
+# The two durable alternatives were weighed and rejected:
+#   * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
+#     a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
+#     of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
+#     would have to be refreshed by hand forever. Too high a price for a replica count.
+#   * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
+#     controller would both own one object and fight continuously. That is the shape of the
+#     `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
+#     both owning one resource, ~2500 events per 10 minutes).
+#
+# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
+# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
+# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
+# and scales back to 2 within one reconcile (~15s by default).
+#
+# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
+# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
+# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
+# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
+#
+# -------------------------------------------------------------------------------------------------
+# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
+# -------------------------------------------------------------------------------------------------
+# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
+# metric to be a valid object, not because coredns is expected to scale on load: measured via
+# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
+# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
+#
+# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
+# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
+# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
+# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
+#
+# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
+# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
+# never balloon DNS pods across the cluster.
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: coredns
+  # kube-system, because that is where k3s puts coredns. This is the only object this component
+  # places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
+  # mutates only the coredns Deployment's scale subresource.
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: coredns
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: coredns
+  minReplicas: 2
+  maxReplicas: 3
+  metrics:
+    - type: Resource
+      resource:
+        name: cpu
+        target:
+          type: Utilization
+          # Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
+          averageUtilization: 80
+  behavior:
+    scaleDown:
+      # 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
+      # queries and every client's negative cache. The HPA should be reluctant in this direction.
+      stabilizationWindowSeconds: 600
+      policies:
+        - type: Pods
+          value: 1
+          periodSeconds: 600
+    scaleUp:
+      # Prompt in the direction that restores redundancy — this is the path taken after a k3s
+      # restart has re-staged `replicas: 1`.
+      stabilizationWindowSeconds: 0
+      policies:
+        - type: Pods
+          value: 2
+          periodSeconds: 60

platform/coredns/kustomization.yaml

@@ -7,3 +7,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - coredns-custom.yaml
+  # HA floor for the coredns Deployment itself — a separate concern from the split-horizon server
+  # block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via
+  # its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning.
+  - hpa.yaml

platform/oauth2-proxy/helmrelease.yaml

@@ -63,6 +63,75 @@ spec:
       # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
       skip-provider-button: "true"
       scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
     # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
     metrics:
       enabled: true