fix(headlamp): per-environment OIDC URLs — prod was dialing DEV's Keycloak
The operator could not log in to Headlamp on prod: Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": dial tcp 192.168.1.53:443: i/o timeout ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24. FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per environment. The Secret keeps only what is genuinely secret or environment-invariant (OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES). Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit env entry silently WINS over the same name from envFrom — so the two sets must not overlap, and they no longer do. The five load-bearing names the chart's $(...) args require (OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered; verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and `helm template` of chart 0.41.0 against the substituted values. Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT `-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via `env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL had been dead weight. The value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts. [parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
d1a6628a97
commit
bfa39bc5d0
4 files changed
+121
-41
No files matched your search
@@ -2,15 +2,27 @@
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
|
||||
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# WHAT THIS IS: the OIDC config Headlamp uses to authenticate against the Keycloak `homelab` realm's
|
||||
# WHAT THIS IS: the SECRET half of Headlamp's OIDC config for the Keycloak `homelab` realm's
|
||||
# `headlamp` client. The Headlamp chart's `config.oidc.externalSecret` (helmrelease.yaml) envFrom's
|
||||
# THIS Secret, so ALL OIDC env (OIDC_CLIENT_ID/OIDC_CLIENT_SECRET/OIDC_ISSUER_URL/OIDC_SCOPES/OIDC_CALLBACK_URL) live here,
|
||||
# SOPS-encrypted, referenced by name only (R-IDP-04). The chart `envFrom`s this Secret VERBATIM, so
|
||||
# the KEYS ARE the container env var names — they MUST be exactly OIDC_CLIENT_ID / OIDC_CLIENT_SECRET
|
||||
# / OIDC_ISSUER_URL / OIDC_SCOPES / OIDC_CALLBACK_URL (the chart's `-oidc-*` args reference those
|
||||
# names via $(...)). There is NO camelCase→OIDC_ mapping; a mismatched key breaks OIDC login.
|
||||
# THIS Secret, so its KEYS ARE the container env var names — the chart renames nothing, and its
|
||||
# `-oidc-*` args reference those names via $(...). They MUST be spelled exactly as below.
|
||||
#
|
||||
# The `clientSecret` value MUST equal the `headlamp-client-secret` placed in
|
||||
# ── THE URLS ARE NOT HERE, AND MUST NOT BE PUT BACK (2026-09-07) ─────────────────────────────
|
||||
# OIDC_ISSUER_URL and OIDC_CALLBACK_URL are ENVIRONMENT-SPECIFIC and NON-SECRET. They live in
|
||||
# `helmrelease.yaml` under `spec.values.env`, written as
|
||||
# https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
# https://headlamp.${BASE_DOMAIN}/oidc-callback
|
||||
# so Flux's postBuild `${BASE_DOMAIN}` substitution resolves them per environment. They used to be
|
||||
# keys in this Secret; substitution CANNOT reach inside ciphertext, so both were frozen at the apex
|
||||
# `home.keithjackson.dev` and PROD pointed Headlamp at DEV's Keycloak (192.168.1.53 — on the dev LAN
|
||||
# behind pfSense, unroutable from prod's 192.168.2.0/24). Every prod login failed with
|
||||
# dial tcp 192.168.1.53:443: i/o timeout on .../.well-known/openid-configuration
|
||||
# Dev only ever looked correct because dev IS the apex. Adding either key back here re-breaks prod.
|
||||
#
|
||||
# ONE SOURCE PER VARIABLE: never define a name in BOTH this Secret and `spec.values.env` — an
|
||||
# explicit `env:` entry silently WINS over the same name from `envFrom`, hiding which one is live.
|
||||
#
|
||||
# The `OIDC_CLIENT_SECRET` value MUST equal the `headlamp-client-secret` placed in
|
||||
# platform/keycloak/keycloak-realm-clients.sops.yaml (which the realm import injects into the
|
||||
# Keycloak `headlamp` client) — same cross-namespace mirror pattern as grafana/gitea (a Secret
|
||||
# cannot be shared across namespaces, so the one value is stored in two encrypted files).
|
||||
@@ -18,7 +30,7 @@
|
||||
# HAND-OFF STEPS (operator):
|
||||
# (a) Use the SAME headlamp-client-secret generated for keycloak-realm-clients (openssl rand -hex 32).
|
||||
# (b) Copy to: platform/headlamp/headlamp-oidc-credentials.sops.yaml
|
||||
# (c) Fill `clientSecret`; encrypt:
|
||||
# (c) Fill `OIDC_CLIENT_SECRET`; encrypt:
|
||||
# sops -e -i platform/headlamp/headlamp-oidc-credentials.sops.yaml
|
||||
# (d) It is already listed under `resources:` in platform/headlamp/kustomization.yaml.
|
||||
# =============================================================================
|
||||
@@ -32,14 +44,12 @@ stringData:
|
||||
# KEY NAMES ARE LOAD-BEARING. The chart does NOT rename these — it `envFrom`s this Secret
|
||||
# verbatim, so each key becomes a container env var of the SAME name, and the chart's OIDC args
|
||||
# reference them by exact name: `-oidc-client-id=$(OIDC_CLIENT_ID)`,
|
||||
# `-oidc-client-secret=$(OIDC_CLIENT_SECRET)`, `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`,
|
||||
# `-oidc-scopes=$(OIDC_SCOPES)`. If a key is misnamed (e.g. camelCase `issuerURL`), the env var
|
||||
# `OIDC_ISSUER_URL` is undefined, K8s leaves the literal string `$(OIDC_ISSUER_URL)` in the arg,
|
||||
# and Headlamp's OIDC discovery fails with `unsupported protocol scheme ""`. Use these EXACT keys.
|
||||
# Non-secret OIDC config (kept in the encrypted Secret because the chart envFrom's a single Secret).
|
||||
# `-oidc-client-secret=$(OIDC_CLIENT_SECRET)`, `-oidc-scopes=$(OIDC_SCOPES)`. If a key is misnamed
|
||||
# (e.g. camelCase `clientID`), the env var is undefined, K8s leaves the LITERAL string
|
||||
# `$(OIDC_CLIENT_ID)` in the arg — it does NOT fail loudly — and OIDC login breaks. Use these
|
||||
# EXACT keys. Environment-INVARIANT, non-secret values stay here simply because the chart wants a
|
||||
# single envFrom source; anything that varies per environment belongs in helmrelease.yaml `env:`.
|
||||
OIDC_CLIENT_ID: headlamp
|
||||
OIDC_ISSUER_URL: https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
OIDC_SCOPES: "openid email profile"
|
||||
OIDC_CALLBACK_URL: https://headlamp.home.keithjackson.dev/oidc-callback
|
||||
# The confidential client secret — MUST match keycloak-realm-clients headlamp-client-secret.
|
||||
OIDC_CLIENT_SECRET: REPLACE-ME
|
||||
Reference in new issue
Block a user