From 960cab937b5a00fd7174d46ac532a0574231c3b8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 27 Aug 2026 18:12:43 -0500 Subject: [PATCH] fix(flux): move oauth2-proxy to a post-platform tier (dependsOn platform-config) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit oauth2-proxy is a leaf forward-auth service that depends on Keycloak + trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform` Flux Kustomization (wait:true), so on a cold start it Failed until those dependencies existed, keeping the whole `platform` tier not-Ready. That in turn blocked `platform-config` — which brings up the very Keycloak instance, wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9). Break the cycle by moving the whole oauth2-proxy component out of the `platform` tier into its own non-gating post-platform Flux Kustomization: - clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy, dependsOn platform-config, wait:false, substituteFrom cluster-vars. - clusters/base/kustomization.yaml: reference the new Kustomization. - platform/kustomization.yaml: drop the oauth2-proxy component (no longer in the gating tier). - platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor so the full component travels together. - platform-config/kustomization.yaml: drop the in-place oauth2-proxy httproute/servicemonitor refs (moved with the controller). - clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that the `platform` Kustomization used to decrypt). Now `platform` reaches Ready without oauth2-proxy; platform-config brings up Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that already exist. wait:false guarantees its health can never gate anything above it. No config of oauth2-proxy itself changed — only its reconciliation tier. Co-Authored-By: Claude Opus 4.8 --- clusters/base/kustomization.yaml | 4 +++ clusters/base/oauth2-proxy.yaml | 44 ++++++++++++++++++++++++ clusters/dev/kustomization.yaml | 14 ++++++++ platform-config/kustomization.yaml | 10 +++--- platform/kustomization.yaml | 10 +++--- platform/oauth2-proxy/kustomization.yaml | 8 +++++ 6 files changed, 81 insertions(+), 9 deletions(-) create mode 100644 clusters/base/oauth2-proxy.yaml diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 2898ed1..cb30402 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -17,4 +17,8 @@ resources: - apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1) # VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves. - apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config) + # oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf + # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating + # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice). + - oauth2-proxy.yaml # Flux Kustomization -> ./platform/oauth2-proxy (dependsOn platform-config) - cluster-vars.yaml # overlay knob surface (base defaults) diff --git a/clusters/base/oauth2-proxy.yaml b/clusters/base/oauth2-proxy.yaml new file mode 100644 index 0000000..ea7fd6e --- /dev/null +++ b/clusters/base/oauth2-proxy.yaml @@ -0,0 +1,44 @@ +--- +# Flux Kustomization: oauth2-proxy forward-auth — reconciles ./platform/oauth2-proxy. +# +# POST-PLATFORM TIER (dependsOn platform-config). oauth2-proxy is a LEAF forward-auth service that +# depends on Keycloak + the trusted wildcard TLS cert + split-horizon DNS — all of which are brought +# up by `platform` (controllers) and `platform-config` (Keycloak instance/realm CRs, the wildcard +# Certificate, cilium L2 VIP). It therefore MUST NOT live inside the gating `platform` tier: on a +# cold start it Fails until its dependencies exist, and while it Failed it kept `platform` not-Ready, +# so `platform-config` (which carries oauth2-proxy's own httproute/servicemonitor AND the Keycloak +# instance it needs) could never reconcile — a self-gating deadlock that bit TWICE (DECISIONS-LOG +# Session 7 + Session 9; roadmap follow-up "oauth2-proxy -> post-platform tier"). Moving it below +# platform-config breaks the cycle: platform reaches Ready without it, platform-config brings up +# Keycloak + TLS + DNS, THEN oauth2-proxy reconciles against dependencies that already exist. +# +# NON-GATING (wait:false): nothing in the graph dependsOn oauth2-proxy (apps waves gate on +# platform-config, not on this), and a forward-auth pod can be legitimately long-not-Ready during a +# cold start while it waits on Keycloak/DNS/TLS — same rationale as apps-vms. wait:false guarantees +# its health can NEVER gate anything, even if a future resource is (mistakenly) pointed dependsOn it. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: oauth2-proxy + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + dependsOn: + - name: platform-config # Keycloak instance/realm + wildcard TLS + split-horizon VIP exist first + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/oauth2-proxy + # R-ENV-02: resolves ${OAUTH2_PROXY_REPLICAS} (dev 1 / prod 2) in the HelmRelease. Same cluster-vars + # ConfigMap the platform tier fed it before the move. + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars + # SOPS decryption for oauth2-proxy-credentials.sops.yaml is layered per-environment by the overlay + # (clusters/dev patches THIS Kustomization by name), identical to how it was decrypted via the + # `platform` Kustomization before the move. See clusters/dev/kustomization.yaml. diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index 38f1fd2..0f36e35 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -63,6 +63,20 @@ patches: provider: sops secretRef: name: sops-age + # oauth2-proxy moved to its own post-platform Flux Kustomization (clusters/base/oauth2-proxy.yaml). + # It carries oauth2-proxy-credentials.sops.yaml, previously decrypted via the `platform` + # Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza. + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: oauth2-proxy + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age - patch: |- apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository diff --git a/platform-config/kustomization.yaml b/platform-config/kustomization.yaml index fbec4e4..c543c13 100644 --- a/platform-config/kustomization.yaml +++ b/platform-config/kustomization.yaml @@ -57,10 +57,10 @@ resources: - ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC) - ../platform/keycloak/httproute.yaml - ../platform/keycloak/servicemonitor.yaml - # oauth2-proxy: forward-auth sign-in/callback ingress + ServiceMonitor (Gateway API / monitoring CRs). - # Re-enabled together with the platform/oauth2-proxy component now Keycloak is live and split-horizon - # DNS resolves to the Proxmox Gateway VIP (192.168.1.53). - - ../platform/oauth2-proxy/httproute.yaml - - ../platform/oauth2-proxy/servicemonitor.yaml + # oauth2-proxy: forward-auth httproute + ServiceMonitor MOVED OUT of platform-config. They now + # travel with the oauth2-proxy controller in its own post-platform Flux Kustomization + # (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). Keeping them here — while the + # controller sat in the gating `platform` tier — was one half of the twice-proven cold-start + # deadlock (platform never Ready -> platform-config never reconciles -> these never apply). # monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR) - ../platform/monitoring/grafana-httproute.yaml diff --git a/platform/kustomization.yaml b/platform/kustomization.yaml index 24f9afd..efb4817 100644 --- a/platform/kustomization.yaml +++ b/platform/kustomization.yaml @@ -26,12 +26,14 @@ resources: # ---- Phase 4 — Observability (creds provisioned via SOPS: grafana-admin, alertmanager-smtp) ---- - monitoring # kube-prometheus-stack + Loki + Alloy + Alertmanager # ---- Phase 6 — Identity / SSO (ADR-0014): foundational IdP, onboarded early. Keycloak Operator + - # CloudNativePG DB + oauth2-proxy forward-auth. Controllers here; the Keycloak/RealmImport/CNPG - # Cluster CRs + ServiceMonitors + HTTPRoutes are in platform-config (dependsOn platform). ---- + # CloudNativePG DB. Controllers here; the Keycloak/RealmImport/CNPG Cluster CRs + ServiceMonitors + # + HTTPRoutes are in platform-config (dependsOn platform). ---- + # oauth2-proxy (forward-auth) is DELIBERATELY NOT here: it depends on Keycloak + trusted TLS + + # split-horizon DNS, so keeping it in this wait:true tier self-gated its own dependencies into a + # cold-start deadlock (twice). It now reconciles in its own post-platform Flux Kustomization + # (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). See DECISIONS-LOG. - cnpg # CloudNativePG operator (DB backend for Keycloak) — ghcr.io images (not Bitnami) - keycloak # Keycloak Operator (quay.io images) + SOPS admin credential - - oauth2-proxy # forward-auth for non-OIDC apps (ADR-0014); OIDC-discovers Keycloak (now live) - # via the split-horizon VIP (GATEWAY_VIP 192.168.1.53) — pre-verified in-cluster. # ---- Later phases: enabled when their phase opens ---- # - velero # Phase 4 (Backup/DR): BLOCKED on OQ-BKP (off-cluster backup target decision) # - image-automation # Phase 4 (Automated updates): Flux image-automation + Renovate diff --git a/platform/oauth2-proxy/kustomization.yaml b/platform/oauth2-proxy/kustomization.yaml index 4a45056..3c80fe0 100644 --- a/platform/oauth2-proxy/kustomization.yaml +++ b/platform/oauth2-proxy/kustomization.yaml @@ -1,4 +1,10 @@ --- +# oauth2-proxy component. Rendered by its OWN post-platform Flux Kustomization +# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config, wait:false) — NOT by the gating +# `platform` tier. The httproute + servicemonitor (Gateway API / monitoring CRs) travel WITH the +# controller here now, rather than being referenced in-place from platform-config: the CRDs they +# need (gateway.networking.k8s.io, monitoring.coreos.com) are registered by platform and platform +# is Ready before this tier's platform-config dependency completes, so ordering holds. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: @@ -6,3 +12,5 @@ resources: - helmrepository.yaml - helmrelease.yaml - oauth2-proxy-credentials.sops.yaml + - httproute.yaml # forward-auth sign-in/callback ingress (was referenced in-place by platform-config) + - servicemonitor.yaml # Prometheus scrape (was referenced in-place by platform-config)