Merge fix/oauth2-proxy-post-platform-tier: move oauth2-proxy to post-platform tier (breaks cold-start deadlock)

2026-08-27 18:15:25 -05:00
commit 0fcbf32742
@@ -17,4 +17,8 @@ resources:
  - apps-wave-2.yaml     # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
  # VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
  - apps-vms.yaml        # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
+  # oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
+  # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
+  # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
+  - oauth2-proxy.yaml    # Flux Kustomization -> ./platform/oauth2-proxy (dependsOn platform-config)
  - cluster-vars.yaml    # overlay knob surface (base defaults)
@@ -0,0 +1,44 @@
+---
+# Flux Kustomization: oauth2-proxy forward-auth — reconciles ./platform/oauth2-proxy.
+#
+# POST-PLATFORM TIER (dependsOn platform-config). oauth2-proxy is a LEAF forward-auth service that
+# depends on Keycloak + the trusted wildcard TLS cert + split-horizon DNS — all of which are brought
+# up by `platform` (controllers) and `platform-config` (Keycloak instance/realm CRs, the wildcard
+# Certificate, cilium L2 VIP). It therefore MUST NOT live inside the gating `platform` tier: on a
+# cold start it Fails until its dependencies exist, and while it Failed it kept `platform` not-Ready,
+# so `platform-config` (which carries oauth2-proxy's own httproute/servicemonitor AND the Keycloak
+# instance it needs) could never reconcile — a self-gating deadlock that bit TWICE (DECISIONS-LOG
+# Session 7 + Session 9; roadmap follow-up "oauth2-proxy -> post-platform tier"). Moving it below
+# platform-config breaks the cycle: platform reaches Ready without it, platform-config brings up
+# Keycloak + TLS + DNS, THEN oauth2-proxy reconciles against dependencies that already exist.
+#
+# NON-GATING (wait:false): nothing in the graph dependsOn oauth2-proxy (apps waves gate on
+# platform-config, not on this), and a forward-auth pod can be legitimately long-not-Ready during a
+# cold start while it waits on Keycloak/DNS/TLS — same rationale as apps-vms. wait:false guarantees
+# its health can NEVER gate anything, even if a future resource is (mistakenly) pointed dependsOn it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: oauth2-proxy
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config   # Keycloak instance/realm + wildcard TLS + split-horizon VIP exist first
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/oauth2-proxy
+  # R-ENV-02: resolves ${OAUTH2_PROXY_REPLICAS} (dev 1 / prod 2) in the HelmRelease. Same cluster-vars
+  # ConfigMap the platform tier fed it before the move.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption for oauth2-proxy-credentials.sops.yaml is layered per-environment by the overlay
+  # (clusters/dev patches THIS Kustomization by name), identical to how it was decrypted via the
+  # `platform` Kustomization before the move. See clusters/dev/kustomization.yaml.
@@ -63,6 +63,20 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
+  # oauth2-proxy moved to its own post-platform Flux Kustomization (clusters/base/oauth2-proxy.yaml).
+  # It carries oauth2-proxy-credentials.sops.yaml, previously decrypted via the `platform`
+  # Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: oauth2-proxy
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
  - patch: |-
      apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
@@ -57,10 +57,10 @@ resources:
  - ../platform/keycloak/k8s-oidc-rbac.yaml   # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
  - ../platform/keycloak/httproute.yaml
  - ../platform/keycloak/servicemonitor.yaml
-  # oauth2-proxy: forward-auth sign-in/callback ingress + ServiceMonitor (Gateway API / monitoring CRs).
-  # Re-enabled together with the platform/oauth2-proxy component now Keycloak is live and split-horizon
-  # DNS resolves to the Proxmox Gateway VIP (192.168.1.53).
-  - ../platform/oauth2-proxy/httproute.yaml
-  - ../platform/oauth2-proxy/servicemonitor.yaml
+  # oauth2-proxy: forward-auth httproute + ServiceMonitor MOVED OUT of platform-config. They now
+  # travel with the oauth2-proxy controller in its own post-platform Flux Kustomization
+  # (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). Keeping them here — while the
+  # controller sat in the gating `platform` tier — was one half of the twice-proven cold-start
+  # deadlock (platform never Ready -> platform-config never reconciles -> these never apply).
  # monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR)
  - ../platform/monitoring/grafana-httproute.yaml
@@ -26,12 +26,14 @@ resources:
  # ---- Phase 4 — Observability (creds provisioned via SOPS: grafana-admin, alertmanager-smtp) ----
  - monitoring         # kube-prometheus-stack + Loki + Alloy + Alertmanager
  # ---- Phase 6 — Identity / SSO (ADR-0014): foundational IdP, onboarded early. Keycloak Operator +
-  #      CloudNativePG DB + oauth2-proxy forward-auth. Controllers here; the Keycloak/RealmImport/CNPG
-  #      Cluster CRs + ServiceMonitors + HTTPRoutes are in platform-config (dependsOn platform). ----
+  #      CloudNativePG DB. Controllers here; the Keycloak/RealmImport/CNPG Cluster CRs + ServiceMonitors
+  #      + HTTPRoutes are in platform-config (dependsOn platform). ----
+  #      oauth2-proxy (forward-auth) is DELIBERATELY NOT here: it depends on Keycloak + trusted TLS +
+  #      split-horizon DNS, so keeping it in this wait:true tier self-gated its own dependencies into a
+  #      cold-start deadlock (twice). It now reconciles in its own post-platform Flux Kustomization
+  #      (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). See DECISIONS-LOG.
  - cnpg               # CloudNativePG operator (DB backend for Keycloak) — ghcr.io images (not Bitnami)
  - keycloak           # Keycloak Operator (quay.io images) + SOPS admin credential
-  - oauth2-proxy       # forward-auth for non-OIDC apps (ADR-0014); OIDC-discovers Keycloak (now live)
-                       # via the split-horizon VIP (GATEWAY_VIP 192.168.1.53) — pre-verified in-cluster.
  # ---- Later phases: enabled when their phase opens ----
  # - velero           # Phase 4 (Backup/DR): BLOCKED on OQ-BKP (off-cluster backup target decision)
  # - image-automation # Phase 4 (Automated updates): Flux image-automation + Renovate
@@ -1,4 +1,10 @@
 ---
+# oauth2-proxy component. Rendered by its OWN post-platform Flux Kustomization
+# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config, wait:false) — NOT by the gating
+# `platform` tier. The httproute + servicemonitor (Gateway API / monitoring CRs) travel WITH the
+# controller here now, rather than being referenced in-place from platform-config: the CRDs they
+# need (gateway.networking.k8s.io, monitoring.coreos.com) are registered by platform and platform
+# is Ready before this tier's platform-config dependency completes, so ordering holds.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
@@ -6,3 +12,5 @@ resources:
  - helmrepository.yaml
  - helmrelease.yaml
  - oauth2-proxy-credentials.sops.yaml
+  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
+  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)