fix(headlamp): per-environment OIDC URLs — prod was dialing DEV's Keycloak

The operator could not log in to Headlamp on prod:

  Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
  dial tcp 192.168.1.53:443: i/o timeout

ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted
headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites
the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen
at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked
correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak
at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24.

FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the
HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per
environment. The Secret keeps only what is genuinely secret or environment-invariant
(OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES).

Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit
env entry silently WINS over the same name from envFrom — so the two sets must not overlap,
and they no longer do. The five load-bearing names the chart's $(...) args require
(OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered;
verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and
`helm template` of chart 0.41.0 against the substituted values.

Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT
`-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via
`env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL
had been dead weight. The value is byte-identical to the single redirectUri registered on the
Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts.

[parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from
k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/headlamp/helmrelease.yaml

@@ -58,16 +58,50 @@ spec:
     config:
       inCluster: true
       oidc:
-        # OIDC config (OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES /
-        # OIDC_CALLBACK_URL) is supplied via an EXISTING SOPS-encrypted Secret consumed by name only
-        # (R-IDP-04) — the chart envFrom's it. The Secret keys ARE the env var names (no renaming),
-        # and the chart's `-oidc-*` args reference them via $(...), e.g.
-        # `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`; a camelCase key would leave that arg unexpanded
-        # and break OIDC login (`unsupported protocol scheme ""`). Nothing sensitive is rendered here.
+        # OIDC config reaches the container as env vars whose NAMES ARE LOAD-BEARING: the chart's
+        # args reference them via $(...) — `-oidc-client-id=$(OIDC_CLIENT_ID)`,
+        # `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-callback-url=$(OIDC_CALLBACK_URL)`, … — so
+        # they MUST be exactly OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES /
+        # OIDC_CALLBACK_URL. A misnamed or absent name does NOT fail: Kubernetes leaves the LITERAL
+        # string `$(OIDC_ISSUER_URL)` in the arg and login dies with `unsupported protocol scheme ""`.
+        #
+        # There are exactly TWO sources, with NO variable served by both (a name defined in `env:`
+        # would silently WIN over the same name coming from `envFrom`, so overlap is forbidden here):
+        #   • envFrom this SOPS Secret → OIDC_CLIENT_ID, OIDC_SCOPES, OIDC_CLIENT_SECRET
+        #   • the `env:` block below   → OIDC_ISSUER_URL, OIDC_CALLBACK_URL
+        # The Secret is consumed by name only (R-IDP-04); nothing sensitive is rendered here.
         externalSecret:
           enabled: true
           name: headlamp-oidc
           hasScopes: true
+    # ── PER-ENVIRONMENT OIDC URLs (2026-09-07 fix) ───────────────────────────────────────────────
+    # These two used to be keys INSIDE headlamp-oidc-credentials.sops.yaml. That was the bug: Flux's
+    # `${BASE_DOMAIN}` postBuild substitution operates on the RENDERED manifest text and can never
+    # reach inside an encrypted Secret, so both were frozen at the apex `home.keithjackson.dev`.
+    # Dev is the apex, so dev looked fine; PROD (BASE_DOMAIN = prod.home.keithjackson.dev) therefore
+    # pointed Headlamp at DEV's Keycloak — 192.168.1.53, on the dev LAN behind pfSense, unroutable
+    # from prod's 192.168.2.0/24 — and every login died with
+    #   dial tcp 192.168.1.53:443: i/o timeout   on .../.well-known/openid-configuration
+    # An issuer URL and a callback URL are PUBLIC. They belong in cleartext, where substitution works.
+    #
+    # [parity] OIDC_ISSUER_URL MUST equal the kube-apiserver's `oidc-issuer-url`, which is an
+    # INSTALL-TIME k3s flag derived from `k3s_base_domain` (infra/ansible roles/k3s_server/defaults +
+    # inventory/group_vars/k3s_<env>/vars.yml), NOT from this ConfigMap. Headlamp forwards the user's
+    # id_token to the apiserver as the bearer, so an issuer mismatch = every request 401s. Both sides
+    # are keyed off the same per-environment domain (BASE_DOMAIN == k3s_base_domain), which is what
+    # keeps them in step — dev home.keithjackson.dev, prod prod.home.keithjackson.dev.
+    #
+    # NOTE: naming OIDC_CALLBACK_URL here also makes the chart EMIT `-oidc-callback-url=$(…)`, which
+    # it previously omitted (the arg is gated on the value being set via `env:`/`config.oidc`, and the
+    # Secret key alone did not trip that gate — so the Secret's callback URL was dead weight). The
+    # value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client
+    # (platform/keycloak/keycloak-realm-import.yaml, also `${BASE_DOMAIN}`-templated), which is the
+    # only URI Keycloak will accept, so making it explicit cannot change behaviour in either env.
+    env:
+      - name: OIDC_ISSUER_URL
+        value: https://keycloak.${BASE_DOMAIN}/realms/homelab
+      - name: OIDC_CALLBACK_URL
+        value: https://headlamp.${BASE_DOMAIN}/oidc-callback
     # PER-USER RBAC: do NOT grant Headlamp's ServiceAccount cluster-admin. User requests are
     # authorized by their forwarded OIDC token against the existing oidc:* group bindings.
     serviceAccount: