fix(flux): move oauth2-proxy to a post-platform tier (dependsOn platform-config)
oauth2-proxy is a leaf forward-auth service that depends on Keycloak + trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform` Flux Kustomization (wait:true), so on a cold start it Failed until those dependencies existed, keeping the whole `platform` tier not-Ready. That in turn blocked `platform-config` — which brings up the very Keycloak instance, wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9). Break the cycle by moving the whole oauth2-proxy component out of the `platform` tier into its own non-gating post-platform Flux Kustomization: - clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy, dependsOn platform-config, wait:false, substituteFrom cluster-vars. - clusters/base/kustomization.yaml: reference the new Kustomization. - platform/kustomization.yaml: drop the oauth2-proxy component (no longer in the gating tier). - platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor so the full component travels together. - platform-config/kustomization.yaml: drop the in-place oauth2-proxy httproute/servicemonitor refs (moved with the controller). - clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that the `platform` Kustomization used to decrypt). Now `platform` reaches Ready without oauth2-proxy; platform-config brings up Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that already exist. wait:false guarantees its health can never gate anything above it. No config of oauth2-proxy itself changed — only its reconciliation tier. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
d34c639269
commit
960cab937b
6 files changed
+81
-9
No files matched your search
@@ -17,4 +17,8 @@ resources:
|
||||
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
||||
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
||||
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
||||
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
||||
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
||||
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
||||
- oauth2-proxy.yaml # Flux Kustomization -> ./platform/oauth2-proxy (dependsOn platform-config)
|
||||
- cluster-vars.yaml # overlay knob surface (base defaults)
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
# Flux Kustomization: oauth2-proxy forward-auth — reconciles ./platform/oauth2-proxy.
|
||||
#
|
||||
# POST-PLATFORM TIER (dependsOn platform-config). oauth2-proxy is a LEAF forward-auth service that
|
||||
# depends on Keycloak + the trusted wildcard TLS cert + split-horizon DNS — all of which are brought
|
||||
# up by `platform` (controllers) and `platform-config` (Keycloak instance/realm CRs, the wildcard
|
||||
# Certificate, cilium L2 VIP). It therefore MUST NOT live inside the gating `platform` tier: on a
|
||||
# cold start it Fails until its dependencies exist, and while it Failed it kept `platform` not-Ready,
|
||||
# so `platform-config` (which carries oauth2-proxy's own httproute/servicemonitor AND the Keycloak
|
||||
# instance it needs) could never reconcile — a self-gating deadlock that bit TWICE (DECISIONS-LOG
|
||||
# Session 7 + Session 9; roadmap follow-up "oauth2-proxy -> post-platform tier"). Moving it below
|
||||
# platform-config breaks the cycle: platform reaches Ready without it, platform-config brings up
|
||||
# Keycloak + TLS + DNS, THEN oauth2-proxy reconciles against dependencies that already exist.
|
||||
#
|
||||
# NON-GATING (wait:false): nothing in the graph dependsOn oauth2-proxy (apps waves gate on
|
||||
# platform-config, not on this), and a forward-auth pod can be legitimately long-not-Ready during a
|
||||
# cold start while it waits on Keycloak/DNS/TLS — same rationale as apps-vms. wait:false guarantees
|
||||
# its health can NEVER gate anything, even if a future resource is (mistakenly) pointed dependsOn it.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config # Keycloak instance/realm + wildcard TLS + split-horizon VIP exist first
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/oauth2-proxy
|
||||
# R-ENV-02: resolves ${OAUTH2_PROXY_REPLICAS} (dev 1 / prod 2) in the HelmRelease. Same cluster-vars
|
||||
# ConfigMap the platform tier fed it before the move.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# SOPS decryption for oauth2-proxy-credentials.sops.yaml is layered per-environment by the overlay
|
||||
# (clusters/dev patches THIS Kustomization by name), identical to how it was decrypted via the
|
||||
# `platform` Kustomization before the move. See clusters/dev/kustomization.yaml.
|
||||
@@ -63,6 +63,20 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# oauth2-proxy moved to its own post-platform Flux Kustomization (clusters/base/oauth2-proxy.yaml).
|
||||
# It carries oauth2-proxy-credentials.sops.yaml, previously decrypted via the `platform`
|
||||
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
|
||||
Reference in new issue
Block a user