From ebe0371b1b968233e48610fb089c6296b56122c7 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 28 Aug 2026 12:01:01 -0500 Subject: [PATCH] platform: move Headlamp out of the gating platform tier + fix -session-ttl crashloop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1 renders a container arg the appVersion v0.40.1 backend lacks; upstream kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy, apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready". Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move all Headlamp resources into a new non-gating Flux Kustomization clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly. The new Kustomization owns the whole platform/headlamp component; its HTTPRoute moves from platform-config into platform/headlamp/kustomization.yaml. Removed headlamp from platform/kustomization.yaml and its httproute from platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall platform/platform-config/apps. Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that same image (appVersion == chart version), so chart+image realign; every arg the chart renders for our values is supported. OIDC config + per-user RBAC (clusterRoleBinding.create:false) unchanged. Gate: `make -C ci gate` PASS. Co-Authored-By: Claude Opus 4.8 --- clusters/base/headlamp.yaml | 45 ++++++++++++++++++++++++++++ clusters/base/kustomization.yaml | 5 ++++ clusters/dev/kustomization.yaml | 14 +++++++++ platform-config/kustomization.yaml | 6 ++-- platform/headlamp/helmrelease.yaml | 10 ++++++- platform/headlamp/kustomization.yaml | 14 ++++++--- platform/kustomization.yaml | 8 +++-- 7 files changed, 92 insertions(+), 10 deletions(-) create mode 100644 clusters/base/headlamp.yaml diff --git a/clusters/base/headlamp.yaml b/clusters/base/headlamp.yaml new file mode 100644 index 0000000..f9aa1ed --- /dev/null +++ b/clusters/base/headlamp.yaml @@ -0,0 +1,45 @@ +--- +# Flux Kustomization: headlamp dashboard — reconciles ./platform/headlamp. +# +# POST-PLATFORM TIER (dependsOn platform-config). Headlamp is a LEAF Kubernetes-dashboard UI that +# depends on Keycloak (native-OIDC login) + the trusted wildcard TLS cert + split-horizon DNS + the +# Gateway API CRDs — all brought up by `platform` (controllers) and `platform-config` (Keycloak +# instance/realm CRs, the wildcard Certificate, cilium L2 VIP, Gateway). It therefore MUST NOT live +# inside the gating `platform` tier: a UI HelmRelease whose image/chart can crashloop (e.g. the +# upstream headlamp-0.40.1 `-session-ttl` chart/image desync, issue #4883) would keep `platform` +# not-Ready, and while `platform` is not-Ready NOTHING downstream reconciles — platform-config, +# oauth2-proxy, every apps-wave and apps-vms all stall on "dependency not ready". That is exactly +# how a single leaf UI stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall"). This is the +# same lesson oauth2-proxy already encodes: a leaf that can be legitimately unhealthy must reconcile +# in its OWN post-platform Kustomization so its health can never gate the graph. +# +# NON-GATING (wait:false): nothing in the graph dependsOn headlamp (apps waves gate on +# platform-config, not on this), and a dashboard pod can be long-not-Ready during a cold start while +# it waits on Keycloak/DNS/TLS — same rationale as oauth2-proxy and apps-vms. wait:false guarantees +# its health can NEVER gate anything, even if a future resource is (mistakenly) pointed dependsOn it. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: headlamp + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + dependsOn: + - name: platform-config # Keycloak instance/realm + wildcard TLS + split-horizon VIP + Gateway first + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/headlamp + # Same cluster-vars ConfigMap the platform tier fed it before the move (headlamp itself references + # no ${VARS} today, but the substitution is retained for parity with the other tiers/future knobs). + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars + # SOPS decryption for headlamp-oidc-credentials.sops.yaml is layered per-environment by the overlay + # (clusters/dev patches THIS Kustomization by name), identical to how it was decrypted via the + # `platform` Kustomization before the move. See clusters/dev/kustomization.yaml. diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index cb30402..295dca1 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -21,4 +21,9 @@ resources: # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice). - oauth2-proxy.yaml # Flux Kustomization -> ./platform/oauth2-proxy (dependsOn platform-config) + # headlamp dashboard — POST-platform tier (dependsOn platform-config, wait:false). A leaf UI whose + # HelmRelease can crashloop (upstream headlamp-0.40.1 `-session-ttl` chart/image desync) must NOT + # sit in the gating `platform` tier, where its unhealth stalled platform-config + oauth2-proxy + + # every apps wave/vm on "dependency not ready" (DECISIONS-LOG: "Headlamp gating stall"). + - headlamp.yaml # Flux Kustomization -> ./platform/headlamp (dependsOn platform-config) - cluster-vars.yaml # overlay knob surface (base defaults) diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index 0f36e35..511c662 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -77,6 +77,20 @@ patches: provider: sops secretRef: name: sops-age + # headlamp moved to its own post-platform Flux Kustomization (clusters/base/headlamp.yaml). + # It carries headlamp-oidc-credentials.sops.yaml, previously decrypted via the `platform` + # Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza. + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: headlamp + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age - patch: |- apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository diff --git a/platform-config/kustomization.yaml b/platform-config/kustomization.yaml index 58810f3..6f7e9e4 100644 --- a/platform-config/kustomization.yaml +++ b/platform-config/kustomization.yaml @@ -64,5 +64,7 @@ resources: # deadlock (platform never Ready -> platform-config never reconciles -> these never apply). # monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR) - ../platform/monitoring/grafana-httproute.yaml - # headlamp: Kubernetes dashboard ingress (Gateway API CR; native-OIDC + per-user RBAC, ADR-0014) - - ../platform/headlamp/httproute.yaml + # headlamp: httproute MOVED OUT of platform-config. It now travels with the headlamp component in + # its own post-platform Flux Kustomization (clusters/base/headlamp.yaml, dependsOn platform-config). + # Keeping it here — while the HelmRelease sat in the gating `platform` tier — meant a headlamp + # crashloop stalled platform-config (and everything below it). See DECISIONS-LOG "Headlamp gating stall". diff --git a/platform/headlamp/helmrelease.yaml b/platform/headlamp/helmrelease.yaml index e155373..b462c6d 100644 --- a/platform/headlamp/helmrelease.yaml +++ b/platform/headlamp/helmrelease.yaml @@ -32,7 +32,15 @@ spec: chart: spec: chart: headlamp - version: ">=0.40.0 <0.41.0" # Renovate maintains; chart appVersion pins the Headlamp image + # 0.41.x, NOT 0.40.x. Chart 0.40.1 shipped the `-session-ttl=86400` container arg BEFORE the + # backend flag existed: its image (appVersion v0.40.1) has no `session-ttl` flag, so the pod + # crashlooped with `flag provided but not defined: -session-ttl` (upstream issue #4883). The + # backend gained `f.Int("session-ttl", …)` in v0.41.0, and chart 0.41.0 pins that same image + # (appVersion == chart version), so chart+image realign: every arg the chart renders for our + # values (-session-ttl + the -oidc-* args from the externalSecret; useCookie stays default-false + # so `-oidc-use-cookie` is NOT emitted) is supported by the v0.41.0 backend. Renovate maintains + # within 0.41.x; the appVersion continues to pin the Headlamp image. + version: ">=0.41.0 <0.42.0" sourceRef: kind: HelmRepository name: headlamp diff --git a/platform/headlamp/kustomization.yaml b/platform/headlamp/kustomization.yaml index a7cba5e..5413d84 100644 --- a/platform/headlamp/kustomization.yaml +++ b/platform/headlamp/kustomization.yaml @@ -1,9 +1,14 @@ --- # Headlamp — Kubernetes dashboard with Keycloak native-OIDC login + per-user RBAC (ADR-0014). -# CONTROLLERS half: the HelmRepository + HelmRelease (chart installs the Deployment/Service/SA), the -# SOPS-encrypted OIDC credential (decrypted by the `platform` Flux Kustomization), and the -# default-deny + Cilium egress/ingress policies. The HTTPRoute is a Gateway API CR and lives in -# platform-config (dependsOn platform) so the Gateway CRDs exist first — same split as grafana. +# Rendered by its OWN post-platform Flux Kustomization (clusters/base/headlamp.yaml, dependsOn +# platform-config, wait:false) — NOT by the gating `platform` tier, where a crashlooping UI +# HelmRelease stalled the whole chain (DECISIONS-LOG "Headlamp gating stall"). This component owns +# the HelmRepository + HelmRelease (chart installs the Deployment/Service/SA), the SOPS-encrypted +# OIDC credential (decrypted per-env by the headlamp Kustomization), the default-deny + Cilium +# policies, AND the HTTPRoute. The httproute (a Gateway API CR) now travels WITH the component here +# rather than being referenced in-place from platform-config: the CRDs it needs +# (gateway.networking.k8s.io) are registered by platform, and platform is Ready before this tier's +# platform-config dependency completes, so ordering holds — same move oauth2-proxy already made. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: headlamp @@ -16,3 +21,4 @@ resources: - headlamp-oidc-credentials.sops.yaml # default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows. - networkpolicy.yaml + - httproute.yaml # dashboard ingress (was referenced in-place by platform-config) diff --git a/platform/kustomization.yaml b/platform/kustomization.yaml index 416a0ff..b0dcedd 100644 --- a/platform/kustomization.yaml +++ b/platform/kustomization.yaml @@ -34,9 +34,11 @@ resources: # (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). See DECISIONS-LOG. - cnpg # CloudNativePG operator (DB backend for Keycloak) — ghcr.io images (not Bitnami) - keycloak # Keycloak Operator (quay.io images) + SOPS admin credential - - headlamp # Kubernetes dashboard — Keycloak native-OIDC login + PER-USER RBAC (ADR-0014). - # HelmRelease + SOPS OIDC credential + default-deny/Cilium policies; its - # HTTPRoute is a Gateway API CR in platform-config (dependsOn platform). + # headlamp (Kubernetes dashboard) is DELIBERATELY NOT here: a leaf UI whose HelmRelease can + # crashloop (upstream headlamp-0.40.1 `-session-ttl` chart/image desync, issue #4883) must not + # sit in this wait:true tier, where its unhealth stalled platform-config + oauth2-proxy + every + # apps wave/vm on "dependency not ready". It now reconciles in its own post-platform Flux + # Kustomization (clusters/base/headlamp.yaml, dependsOn platform-config). See DECISIONS-LOG. # ---- Later phases: enabled when their phase opens ---- # - velero # Phase 4 (Backup/DR): BLOCKED on OQ-BKP (off-cluster backup target decision) # - image-automation # Phase 4 (Automated updates): Flux image-automation + Renovate