From bfa39bc5d05eb0e324754f36fb77186b9b2b73bb Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 02:06:38 -0500 Subject: [PATCH 1/3] =?UTF-8?q?fix(headlamp):=20per-environment=20OIDC=20U?= =?UTF-8?q?RLs=20=E2=80=94=20prod=20was=20dialing=20DEV's=20Keycloak?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operator could not log in to Headlamp on prod: Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": dial tcp 192.168.1.53:443: i/o timeout ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24. FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per environment. The Secret keeps only what is genuinely secret or environment-invariant (OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES). Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit env entry silently WINS over the same name from envFrom — so the two sets must not overlap, and they no longer do. The five load-bearing names the chart's $(...) args require (OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered; verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and `helm template` of chart 0.41.0 against the substituted values. Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT `-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via `env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL had been dead weight. The value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts. [parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses. Co-Authored-By: Claude Opus 5 --- platform/headlamp/README.md | 54 ++++++++++++++++--- .../headlamp-oidc-credentials.sops.yaml | 22 ++++---- ...eadlamp-oidc-credentials.sops.yaml.example | 40 ++++++++------ platform/headlamp/helmrelease.yaml | 46 +++++++++++++--- 4 files changed, 121 insertions(+), 41 deletions(-) diff --git a/platform/headlamp/README.md b/platform/headlamp/README.md index f2972aa..9cd5b3a 100644 --- a/platform/headlamp/README.md +++ b/platform/headlamp/README.md @@ -1,6 +1,6 @@ # Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC) -Headlamp is served at `https://headlamp.home.keithjackson.dev` behind the shared Gateway +Headlamp is served at `https://headlamp.${BASE_DOMAIN}` behind the shared Gateway (wildcard TLS), with **native Keycloak OIDC login** and **per-user RBAC** (ADR-0014, docs/15). ## How per-user RBAC works (end to end) @@ -33,6 +33,49 @@ carry `aud=[headlamp, kubernetes]`: This keeps the apiserver OIDC config untouched (preferred over widening `--api-audiences`). +## Where the OIDC config comes from (two sources, no overlap) + +The container needs five env vars, and their **names are load-bearing**: the chart's args reference +them via `$(...)` (`-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-client-id=$(OIDC_CLIENT_ID)`, …). +They must be exactly `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_ISSUER_URL`, `OIDC_SCOPES`, +`OIDC_CALLBACK_URL` — **not** camelCase. A misnamed or missing name does **not** fail loudly: +Kubernetes leaves the literal string `$(OIDC_ISSUER_URL)` in the argument and login dies with +`unsupported protocol scheme ""`. + +| Env var | Source | Why | +|---|---|---| +| `OIDC_CLIENT_ID`, `OIDC_SCOPES`, `OIDC_CLIENT_SECRET` | `headlamp-oidc-credentials.sops.yaml`, `envFrom`'d by `config.oidc.externalSecret` | secret, or environment-invariant | +| `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` | `helmrelease.yaml` → `spec.values.env`, templated `${BASE_DOMAIN}` | **environment-specific and public** | + +**Exactly one source per variable.** Never define the same name in both places: an explicit `env:` +entry silently overrides the same name arriving via `envFrom`, so an overlap hides which value is +actually live. + +### Why the URLs are not in the Secret (the 2026-09-07 prod outage) + +Flux's `${BASE_DOMAIN}` postBuild substitution runs over the **rendered manifest text**; it can never +reach inside an encrypted Secret. `OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` used to be keys in +`headlamp-oidc-credentials.sops.yaml`, so both were frozen at the apex `home.keithjackson.dev`. +Dev **is** the apex, so dev looked fine — but prod (`BASE_DOMAIN = prod.home.keithjackson.dev`) was +pointed at **dev's** Keycloak, `192.168.1.53`, which sits on the dev LAN behind pfSense and is +unroutable from prod's `192.168.2.0/24`. Every prod login failed with: + +``` +Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": +dial tcp 192.168.1.53:443: i/o timeout +``` + +An issuer URL and a callback URL are public, so they now live in cleartext in `helmrelease.yaml` +where substitution works. Only the client secret stays encrypted. The same root cause and the same +fix apply to Semaphore (`catalog/semaphore.yaml`, `SEMAPHORE_OIDC_PROVIDERS`). + +**[parity] `OIDC_ISSUER_URL` must equal the kube-apiserver's `oidc-issuer-url`**, which is an +install-time k3s flag set from `k3s_base_domain` (`infra/ansible/roles/k3s_server/defaults/main.yml`, +overridden per inventory group), *not* from `cluster-vars`. Headlamp forwards the user's `id_token` +to the apiserver, so a mismatch 401s every request. Both derive from the same per-environment +domain — dev `home.keithjackson.dev`, prod `prod.home.keithjackson.dev` — which is what keeps them +in step; changing one without the other breaks login. + ## Secrets - The confidential client secret lives SOPS-encrypted in @@ -41,11 +84,6 @@ This keeps the apiserver OIDC config untouched (preferred over widening `--api-a placeholder) and mirrored in this namespace's `headlamp-oidc-credentials.sops.yaml` (`OIDC_CLIENT_SECRET`), which the chart `config.oidc.externalSecret` envFrom's. Same cross-namespace mirror pattern as grafana/gitea. -- **Key names are load-bearing.** The chart `envFrom`s this Secret verbatim, so its keys become the - container env vars referenced by the chart's OIDC args (`-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, - etc.). They must be exactly `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_ISSUER_URL`, `OIDC_SCOPES` - (and `OIDC_CALLBACK_URL`) — **not** camelCase. A misnamed key leaves `$(OIDC_ISSUER_URL)` - unexpanded and OIDC login fails with `unsupported protocol scheme ""`. ## Break-glass @@ -58,6 +96,6 @@ access if Keycloak/OIDC is down — the standard break-glass path. |---|---|---| | `namespace.yaml` | platform | `headlamp` namespace (`gateway-access=shared`; platform, not app-namespace) | | `helmrepository.yaml` / `helmrelease.yaml` | platform | Official `kubernetes-sigs/headlamp` chart + OIDC/RBAC values | -| `headlamp-oidc-credentials.sops.yaml` | platform | SOPS OIDC config the chart envFrom's (decrypted by the `platform` Flux Kustomization) | +| `headlamp-oidc-credentials.sops.yaml` | platform | SOPS OIDC **secret** the chart envFrom's — client id/scopes/secret only; the per-env URLs live in `helmrelease.yaml` `spec.values.env` | | `networkpolicy.yaml` | platform | default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows | -| `httproute.yaml` | platform-config | Gateway API HTTPRoute for `headlamp.home.keithjackson.dev` (needs Gateway CRDs) | +| `httproute.yaml` | platform-config | Gateway API HTTPRoute for `headlamp.${BASE_DOMAIN}` (needs Gateway CRDs) | diff --git a/platform/headlamp/headlamp-oidc-credentials.sops.yaml b/platform/headlamp/headlamp-oidc-credentials.sops.yaml index c266222..b00dd88 100644 --- a/platform/headlamp/headlamp-oidc-credentials.sops.yaml +++ b/platform/headlamp/headlamp-oidc-credentials.sops.yaml @@ -5,11 +5,9 @@ metadata: namespace: headlamp type: Opaque stringData: - OIDC_CLIENT_ID: ENC[AES256_GCM,data:dXAzoFfar4A=,iv:dHOJFcS6tpvtJes1c++nU6BbHyRlmGuLpncucbkJ4xA=,tag:lFI+KJPid2f/2r/ZjYlFTg==,type:str] - OIDC_ISSUER_URL: ENC[AES256_GCM,data:Er1ljhH62WahtEJkNvxk1BA2jz28OOCZmmyjkbVN82e/TFM7O+FB/FPjWUaW3lhd+dK16ro=,iv:Z3aP7+T+ytItYpbxjoYymLNGa7m8KGic4WqIQrrBSGE=,tag:/Rvhp5+r7AXIj2uDgL/dPg==,type:str] - OIDC_SCOPES: ENC[AES256_GCM,data:WoGaQKY9BFsi5/kDlRMXatINtmE=,iv:kTN+C176I8o5ftzZiKR10s9gl9VvBsOCJhFIZzj9gm4=,tag:ZyRIykD1h5YBq4grd6FB/w==,type:str] - OIDC_CALLBACK_URL: ENC[AES256_GCM,data:SZdpxEuW1BJccLyRGjuIDWBu6KpySARv/6M0mWtzc9jS+v64Lgr4VWKwgW0tdZrH5jM9rw==,iv:UbLwgfz7J0SIWbcQ4csqxv5B3enTmfHJzSDw/lmqAEI=,tag:lge2AuoQXK/u3uZbWvGnLQ==,type:str] - OIDC_CLIENT_SECRET: ENC[AES256_GCM,data:r0pb73gat8E7BFLPbdhguLh8G5PUT7PlJJeIcv7QPdMnN4iIB8MedMwxd77DkbxHSR7bav2ohQczjnbJBA4tPA==,iv:O0RbMFz4UD1sy8mbY7SXWCSGNF+lapDTL7tpphEcx7c=,tag:bEyuEs/lsK28fJeYDpYiOg==,type:str] + OIDC_CLIENT_ID: ENC[AES256_GCM,data:FCW6yV5oUCM=,iv:+oWwATPxrukg7BbNDraaWhBoLmRsAAlF/HwlyYtLBgI=,tag:aoEMlyxyXLeiLak+i3Suww==,type:str] + OIDC_SCOPES: ENC[AES256_GCM,data:k9jHaTSEQ7zl0gmK2zTW7AAbQ8c=,iv:cOGN1Oi/E9tTSZqNKY6i35wWk6fXatlYNRRhtVQrl1E=,tag:Q3I2pJ/a6ESX/ox2VvNDiA==,type:str] + OIDC_CLIENT_SECRET: ENC[AES256_GCM,data:qoWJYdYx5bZztHJRYlGgxVRy/n9izkgffoGuQwePWtV6YrQpxZGgS3sak8cP6tt5M+3ZkFSrysxgcqcjrzJFuA==,iv:ZtEfWbPOucVnYv3x107XEjZ8Qt7HXBpwTlSuoxv+x5I=,tag:PXuEyhIoPOxo/Aw6hEhfKw==,type:str] sops: kms: [] gcp_kms: [] @@ -19,14 +17,14 @@ sops: - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1UDV1RzRWeGZvdWxFQlgz - dCtjTU8wZzR5Tks5dmYreFliREZGaDdXMVVjCkVPY2lZby9LNzJMelJkTlE1Yks2 - cXN5YmhlaUtwN3ZjQUo5R0xnQWNQYm8KLS0tIDBMSUMxZ1pHaDRnUWs5RGtRNlhI - N0I4ekhqMWVWUFZoeGU4SERpYXVDQUkKUUwY6ZxEiWGdqV8s3wbnbPF9SkZJoxfy - 3MkNQ7rZHSILFTPVaZSErNTI21vqQr3lG24+yKLfNhAZBNyB5lvedQ== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFdDZ6Zm9hSW9NY1Mrd0FO + NXQxdC9NWXlrd2VXUjBYeGVLREJjOU56dmdJCk15NzlCdFRaWDdvUWVTcWtRc3hj + eVJCNVdYSFZQU0V4Q0txTzhEMHVJMUEKLS0tIG5kamFsU3RwTWp6MWRlNTlQRlE3 + MFJFZ2NvUXAvc3J6NUl2b0NHNUVDYTQKb7tibY/OHM4FRfQ2dzTnIRTIgWG8gddU + lFJCZG03U79MEG/UeottO4uNJ3jSXImXpysXSHnH3o5muNvIuEN9rw== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-08-29T20:04:33Z" - mac: ENC[AES256_GCM,data:Haj8wMqDuIdtBopP02L1Z63+T8f0fcuSDpF5jnjgCB1+uzX+d4ylqpcVP5r8Ug8M/2ay2nYWoJBWzL0Y1iHcnDP7/T60M8hT46cBYPrDh3MqY0rWF5GwpjJxL9tStoSlDTYICeilAPqo2Vw4UM1CqnZv+th40rz5/R4MwRsU04w=,iv:5HPvtnF6nnDYDaT84r1JWf+ZH1X5/L1Aiim2lgOucH4=,tag:VM/2FXNW7sdjCk/k2AVl+g==,type:str] + lastmodified: "2026-09-07T07:04:07Z" + mac: ENC[AES256_GCM,data:hGJmhdXg0XaaCh6YRI7Wj1TlnjFtckASY4OSyfEgLM8tbwPoElXLir5HQJz7F19uPirbSslVuG8+s58Zvv4/JG/Qki9tQyTItJzbkFVg7iXnXOcYwxW8vw+P26NeE84CYXCIGGunzKZuY7/6ftul8czt/3p83I0X4Y4YzbttaGs=,iv:i2wY7tPaeizv1k7O7b9bMSdbGABFDri5WdDsOxGBkOw=,tag:QI1b2rYgeme26NoGBm3GEA==,type:str] pgp: [] encrypted_regex: ^(data|stringData)$ version: 3.9.4 diff --git a/platform/headlamp/headlamp-oidc-credentials.sops.yaml.example b/platform/headlamp/headlamp-oidc-credentials.sops.yaml.example index b35f57e..cc3a58b 100644 --- a/platform/headlamp/headlamp-oidc-credentials.sops.yaml.example +++ b/platform/headlamp/headlamp-oidc-credentials.sops.yaml.example @@ -2,15 +2,27 @@ # TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never # auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF. # -# WHAT THIS IS: the OIDC config Headlamp uses to authenticate against the Keycloak `homelab` realm's +# WHAT THIS IS: the SECRET half of Headlamp's OIDC config for the Keycloak `homelab` realm's # `headlamp` client. The Headlamp chart's `config.oidc.externalSecret` (helmrelease.yaml) envFrom's -# THIS Secret, so ALL OIDC env (OIDC_CLIENT_ID/OIDC_CLIENT_SECRET/OIDC_ISSUER_URL/OIDC_SCOPES/OIDC_CALLBACK_URL) live here, -# SOPS-encrypted, referenced by name only (R-IDP-04). The chart `envFrom`s this Secret VERBATIM, so -# the KEYS ARE the container env var names — they MUST be exactly OIDC_CLIENT_ID / OIDC_CLIENT_SECRET -# / OIDC_ISSUER_URL / OIDC_SCOPES / OIDC_CALLBACK_URL (the chart's `-oidc-*` args reference those -# names via $(...)). There is NO camelCase→OIDC_ mapping; a mismatched key breaks OIDC login. +# THIS Secret, so its KEYS ARE the container env var names — the chart renames nothing, and its +# `-oidc-*` args reference those names via $(...). They MUST be spelled exactly as below. # -# The `clientSecret` value MUST equal the `headlamp-client-secret` placed in +# ── THE URLS ARE NOT HERE, AND MUST NOT BE PUT BACK (2026-09-07) ───────────────────────────── +# OIDC_ISSUER_URL and OIDC_CALLBACK_URL are ENVIRONMENT-SPECIFIC and NON-SECRET. They live in +# `helmrelease.yaml` under `spec.values.env`, written as +# https://keycloak.${BASE_DOMAIN}/realms/homelab +# https://headlamp.${BASE_DOMAIN}/oidc-callback +# so Flux's postBuild `${BASE_DOMAIN}` substitution resolves them per environment. They used to be +# keys in this Secret; substitution CANNOT reach inside ciphertext, so both were frozen at the apex +# `home.keithjackson.dev` and PROD pointed Headlamp at DEV's Keycloak (192.168.1.53 — on the dev LAN +# behind pfSense, unroutable from prod's 192.168.2.0/24). Every prod login failed with +# dial tcp 192.168.1.53:443: i/o timeout on .../.well-known/openid-configuration +# Dev only ever looked correct because dev IS the apex. Adding either key back here re-breaks prod. +# +# ONE SOURCE PER VARIABLE: never define a name in BOTH this Secret and `spec.values.env` — an +# explicit `env:` entry silently WINS over the same name from `envFrom`, hiding which one is live. +# +# The `OIDC_CLIENT_SECRET` value MUST equal the `headlamp-client-secret` placed in # platform/keycloak/keycloak-realm-clients.sops.yaml (which the realm import injects into the # Keycloak `headlamp` client) — same cross-namespace mirror pattern as grafana/gitea (a Secret # cannot be shared across namespaces, so the one value is stored in two encrypted files). @@ -18,7 +30,7 @@ # HAND-OFF STEPS (operator): # (a) Use the SAME headlamp-client-secret generated for keycloak-realm-clients (openssl rand -hex 32). # (b) Copy to: platform/headlamp/headlamp-oidc-credentials.sops.yaml -# (c) Fill `clientSecret`; encrypt: +# (c) Fill `OIDC_CLIENT_SECRET`; encrypt: # sops -e -i platform/headlamp/headlamp-oidc-credentials.sops.yaml # (d) It is already listed under `resources:` in platform/headlamp/kustomization.yaml. # ============================================================================= @@ -32,14 +44,12 @@ stringData: # KEY NAMES ARE LOAD-BEARING. The chart does NOT rename these — it `envFrom`s this Secret # verbatim, so each key becomes a container env var of the SAME name, and the chart's OIDC args # reference them by exact name: `-oidc-client-id=$(OIDC_CLIENT_ID)`, - # `-oidc-client-secret=$(OIDC_CLIENT_SECRET)`, `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, - # `-oidc-scopes=$(OIDC_SCOPES)`. If a key is misnamed (e.g. camelCase `issuerURL`), the env var - # `OIDC_ISSUER_URL` is undefined, K8s leaves the literal string `$(OIDC_ISSUER_URL)` in the arg, - # and Headlamp's OIDC discovery fails with `unsupported protocol scheme ""`. Use these EXACT keys. - # Non-secret OIDC config (kept in the encrypted Secret because the chart envFrom's a single Secret). + # `-oidc-client-secret=$(OIDC_CLIENT_SECRET)`, `-oidc-scopes=$(OIDC_SCOPES)`. If a key is misnamed + # (e.g. camelCase `clientID`), the env var is undefined, K8s leaves the LITERAL string + # `$(OIDC_CLIENT_ID)` in the arg — it does NOT fail loudly — and OIDC login breaks. Use these + # EXACT keys. Environment-INVARIANT, non-secret values stay here simply because the chart wants a + # single envFrom source; anything that varies per environment belongs in helmrelease.yaml `env:`. OIDC_CLIENT_ID: headlamp - OIDC_ISSUER_URL: https://keycloak.home.keithjackson.dev/realms/homelab OIDC_SCOPES: "openid email profile" - OIDC_CALLBACK_URL: https://headlamp.home.keithjackson.dev/oidc-callback # The confidential client secret — MUST match keycloak-realm-clients headlamp-client-secret. OIDC_CLIENT_SECRET: REPLACE-ME diff --git a/platform/headlamp/helmrelease.yaml b/platform/headlamp/helmrelease.yaml index 4021de4..9aa5156 100644 --- a/platform/headlamp/helmrelease.yaml +++ b/platform/headlamp/helmrelease.yaml @@ -58,16 +58,50 @@ spec: config: inCluster: true oidc: - # OIDC config (OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES / - # OIDC_CALLBACK_URL) is supplied via an EXISTING SOPS-encrypted Secret consumed by name only - # (R-IDP-04) — the chart envFrom's it. The Secret keys ARE the env var names (no renaming), - # and the chart's `-oidc-*` args reference them via $(...), e.g. - # `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`; a camelCase key would leave that arg unexpanded - # and break OIDC login (`unsupported protocol scheme ""`). Nothing sensitive is rendered here. + # OIDC config reaches the container as env vars whose NAMES ARE LOAD-BEARING: the chart's + # args reference them via $(...) — `-oidc-client-id=$(OIDC_CLIENT_ID)`, + # `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-callback-url=$(OIDC_CALLBACK_URL)`, … — so + # they MUST be exactly OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES / + # OIDC_CALLBACK_URL. A misnamed or absent name does NOT fail: Kubernetes leaves the LITERAL + # string `$(OIDC_ISSUER_URL)` in the arg and login dies with `unsupported protocol scheme ""`. + # + # There are exactly TWO sources, with NO variable served by both (a name defined in `env:` + # would silently WIN over the same name coming from `envFrom`, so overlap is forbidden here): + # • envFrom this SOPS Secret → OIDC_CLIENT_ID, OIDC_SCOPES, OIDC_CLIENT_SECRET + # • the `env:` block below → OIDC_ISSUER_URL, OIDC_CALLBACK_URL + # The Secret is consumed by name only (R-IDP-04); nothing sensitive is rendered here. externalSecret: enabled: true name: headlamp-oidc hasScopes: true + # ── PER-ENVIRONMENT OIDC URLs (2026-09-07 fix) ─────────────────────────────────────────────── + # These two used to be keys INSIDE headlamp-oidc-credentials.sops.yaml. That was the bug: Flux's + # `${BASE_DOMAIN}` postBuild substitution operates on the RENDERED manifest text and can never + # reach inside an encrypted Secret, so both were frozen at the apex `home.keithjackson.dev`. + # Dev is the apex, so dev looked fine; PROD (BASE_DOMAIN = prod.home.keithjackson.dev) therefore + # pointed Headlamp at DEV's Keycloak — 192.168.1.53, on the dev LAN behind pfSense, unroutable + # from prod's 192.168.2.0/24 — and every login died with + # dial tcp 192.168.1.53:443: i/o timeout on .../.well-known/openid-configuration + # An issuer URL and a callback URL are PUBLIC. They belong in cleartext, where substitution works. + # + # [parity] OIDC_ISSUER_URL MUST equal the kube-apiserver's `oidc-issuer-url`, which is an + # INSTALL-TIME k3s flag derived from `k3s_base_domain` (infra/ansible roles/k3s_server/defaults + + # inventory/group_vars/k3s_/vars.yml), NOT from this ConfigMap. Headlamp forwards the user's + # id_token to the apiserver as the bearer, so an issuer mismatch = every request 401s. Both sides + # are keyed off the same per-environment domain (BASE_DOMAIN == k3s_base_domain), which is what + # keeps them in step — dev home.keithjackson.dev, prod prod.home.keithjackson.dev. + # + # NOTE: naming OIDC_CALLBACK_URL here also makes the chart EMIT `-oidc-callback-url=$(…)`, which + # it previously omitted (the arg is gated on the value being set via `env:`/`config.oidc`, and the + # Secret key alone did not trip that gate — so the Secret's callback URL was dead weight). The + # value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client + # (platform/keycloak/keycloak-realm-import.yaml, also `${BASE_DOMAIN}`-templated), which is the + # only URI Keycloak will accept, so making it explicit cannot change behaviour in either env. + env: + - name: OIDC_ISSUER_URL + value: https://keycloak.${BASE_DOMAIN}/realms/homelab + - name: OIDC_CALLBACK_URL + value: https://headlamp.${BASE_DOMAIN}/oidc-callback # PER-USER RBAC: do NOT grant Headlamp's ServiceAccount cluster-admin. User requests are # authorized by their forwarded OIDC token against the existing oidc:* group bindings. serviceAccount: From 6548f9a89c12b489750418f3f941a0f59d413b43 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 02:06:52 -0500 Subject: [PATCH 2/3] =?UTF-8?q?fix(semaphore):=20per-environment=20OIDC=20?= =?UTF-8?q?URLs=20=E2=80=94=20same=20latent=20break=20as=20Headlamp?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext — provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the Headlamp login timeout, not yet reported only because nobody had tried prod SSO. FIX: split the blob. The URLs are public and now live as a plaintext literal in catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like every other hostname. Only the client secret stays encrypted, in its own key `oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion: $(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared immediately above it (order is load-bearing — the kubelet resolves env in list order and leaves a forward reference as literal text). Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret (the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} — dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev. apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical `catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`. Co-Authored-By: Claude Opus 5 --- .../semaphore-bootstrap-secret.sops.yaml | 33 +++++++++------ apps/wave-1/semaphore.yaml | 6 ++- catalog/semaphore.yaml | 42 ++++++++++++++----- 3 files changed, 56 insertions(+), 25 deletions(-) diff --git a/apps/wave-1/semaphore-bootstrap-secret.sops.yaml b/apps/wave-1/semaphore-bootstrap-secret.sops.yaml index 490de04..20124f5 100644 --- a/apps/wave-1/semaphore-bootstrap-secret.sops.yaml +++ b/apps/wave-1/semaphore-bootstrap-secret.sops.yaml @@ -5,6 +5,15 @@ # credential from its own namespace (a Pod cannot read a Secret in another namespace). # access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys / # credentials; MUST stay stable across restarts or stored secrets become undecryptable. +# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS +# `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import +# injects that copy into the client), same cross-namespace mirror pattern as db-password above. +# It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE +# SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect +# URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them, +# so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in +# catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value +# back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted. # Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml apiVersion: v1 kind: Secret @@ -15,11 +24,11 @@ metadata: app.kubernetes.io/name: semaphore type: Opaque stringData: - admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str] - admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str] - db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str] - access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str] - oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str] + admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str] + admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str] + db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str] + access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str] + oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str] sops: kms: [] gcp_kms: [] @@ -29,14 +38,14 @@ sops: - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP - S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO - MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX - elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka - w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2 + VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt + cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4 + RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ + 9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-08-27T22:12:32Z" - mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str] + lastmodified: "2026-09-07T07:01:50Z" + mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str] pgp: [] encrypted_regex: ^(data|stringData)$ version: 3.9.4 diff --git a/apps/wave-1/semaphore.yaml b/apps/wave-1/semaphore.yaml index d29a2e6..010e334 100644 --- a/apps/wave-1/semaphore.yaml +++ b/apps/wave-1/semaphore.yaml @@ -118,11 +118,13 @@ spec: key: access-key-encryption - name: SEMAPHORE_PORT value: :3000 - - name: SEMAPHORE_OIDC_PROVIDERS + - name: SEMAPHORE_OIDC_CLIENT_SECRET valueFrom: secretKeyRef: name: semaphore-bootstrap - key: oidc-providers-json + key: oidc-client-secret + - name: SEMAPHORE_OIDC_PROVIDERS + value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}' volumeMounts: - name: config mountPath: /etc/semaphore diff --git a/catalog/semaphore.yaml b/catalog/semaphore.yaml index 34ae9b4..a20855b 100644 --- a/catalog/semaphore.yaml +++ b/catalog/semaphore.yaml @@ -78,16 +78,34 @@ env: # (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single # JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery), # client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's - # token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the - # semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in - # keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a - # kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision. - # The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). --- - - name: SEMAPHORE_OIDC_PROVIDERS + # token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / + # SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15). + # + # PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted + # semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential + # client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the + # redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT + # reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN = + # prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's + # 192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see + # platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex. + # + # THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets + # ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential + # client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back + # in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env + # var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So + # SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list + # order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is + # fail-loud in the way that matters: if the Secret key is missing the Pod never starts + # (CreateContainerConfigError), rather than silently authenticating against nothing. --- + - name: SEMAPHORE_OIDC_CLIENT_SECRET valueFrom: secretKeyRef: name: semaphore-bootstrap - key: oidc-providers-json + key: oidc-client-secret + - name: SEMAPHORE_OIDC_PROVIDERS + value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}' # The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper), # and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC # is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes @@ -117,10 +135,12 @@ auth: groups: - platform-admins # access scoped to an authorized group (R-IDP-06) # WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm- - # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted - # in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for - # authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage - # never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here. + # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob + # (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential + # client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`. + # The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the + # break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat"). + # No secret is embedded here. update: channel: stable # R-UPD-01 range: ">=2.19.0 <3.0.0" # stay on the 2.x stable line From e65a76dc5118ba2fac6419cfcc2e9d37a5b894e8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 02:10:10 -0500 Subject: [PATCH 3/3] docs(log): record the prod OIDC-URL outage and the rule it establishes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Non-secret, environment-specific values must not live inside an encrypted Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and cannot reach inside ciphertext, so such a value silently freezes at whatever one environment happened to need. Dev was correct only by coincidence (dev IS the apex); prod was wrong from its first reconcile, for both Headlamp and Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no unresolved ${...} to complain about. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 89 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index cf00eac..f9aedc0 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1758,3 +1758,92 @@ into a transient hard Kustomization failure. - **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway. Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6. + +## 2026-09-07 — LIVE BUG: prod's OIDC clients were pointed at DEV's Keycloak. Secrets are a substitution blind spot. + +The operator could not log in to Headlamp on prod: + +``` +Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": +dial tcp 192.168.1.53:443: i/o timeout +``` + +`keycloak.home.keithjackson.dev` is **dev's** Keycloak. It resolves to `192.168.1.53`, on the dev +LAN behind pfSense; prod lives on `192.168.2.0/24` (pfSense's WAN) and cannot route there, so the +discovery fetch simply hung until it timed out. + +### Root cause — the class, not the instance + +**Flux's `${BASE_DOMAIN}` postBuild substitution rewrites the RENDERED MANIFEST TEXT. It can never +reach inside a SOPS-encrypted Secret.** Environment-specific URLs had been stored *inside* two +encrypted Secrets purely because it was convenient to keep one blob per app: + +| File | Buried, un-substitutable values | +|---|---| +| `platform/headlamp/headlamp-oidc-credentials.sops.yaml` | `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` | +| `apps/wave-1/semaphore-bootstrap-secret.sops.yaml` (`oidc-providers-json`) | the realm issuer + the OIDC redirect URI | + +Both were frozen at the apex `home.keithjackson.dev`. **Dev IS the apex, so dev was correct by +coincidence** — the values were never per-environment, they just happened to match one environment. +Prod was wrong from the first reconcile. Semaphore on prod was broken the same way and nobody had +noticed, because nobody had tried prod SSO yet. + +This is the same failure shape as the 2026-09-01 nextcloud guard: **a value that silently resolves +to something plausible is worse than one that fails.** `StrictPostBuildSubstitutions` cannot catch +it either — there is no `${...}` left unresolved to complain about; the ciphertext never contained +one. + +### The rule this establishes + +> **Non-secret, environment-specific values MUST NOT live inside an encrypted Secret.** Issuer URLs, +> callback URLs, redirect URIs and hostnames are public. Encrypt the secret half only, and template +> the rest where `${BASE_DOMAIN}` can reach it. + +### The fix + +- **Headlamp** — `OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` moved to the HelmRelease's + `spec.values.env`, templated `${BASE_DOMAIN}`. The Secret keeps `OIDC_CLIENT_SECRET` + + the environment-invariant `OIDC_CLIENT_ID` / `OIDC_SCOPES`, still `envFrom`'d. + **Exactly one source per variable** — an explicit `env:` entry silently outranks the same name + arriving via `envFrom`, so the two sets deliberately do not overlap. + Side effect, intended: naming `OIDC_CALLBACK_URL` in `env:` makes chart 0.41.0 finally emit + `-oidc-callback-url=$(OIDC_CALLBACK_URL)`; it gates that arg on `env:`/`config.oidc`, which a + Secret key alone never tripped — so the Secret's callback URL had been dead weight all along. +- **Semaphore** — the `SEMAPHORE_OIDC_PROVIDERS` JSON is now a plaintext literal in + `catalog/semaphore.yaml` (re-rendered to the `apps/wave-1` golden). Only the client secret stays + encrypted, in its own key `oidc-client-secret`, and the **kubelet** splices it back via + dependent-env-var expansion: `$(SEMAPHORE_OIDC_CLIENT_SECRET)` resolves against the `secretKeyRef` + env var declared immediately above it. **Order is load-bearing** — the kubelet resolves `env` in + list order and leaves a forward reference as literal text. `flux envsubst` ignores `$(...)`, so + the two expansion layers do not collide (the `$${VAR}` escape in `apps/wave-1/nextcloud.yaml` + documents the one form that DOES need escaping). + +### Proof + +Rendered per overlay (`kubectl kustomize --load-restrictor LoadRestrictionsNone`, then +`flux envsubst --strict` with that overlay's `cluster-vars`), plus `helm template` of headlamp +0.41.0 against the substituted values: + +| | dev | prod | +|---|---|---| +| Headlamp `OIDC_ISSUER_URL` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` | +| Headlamp `OIDC_CALLBACK_URL` | `https://headlamp.home.keithjackson.dev/oidc-callback` | `https://headlamp.prod.home.keithjackson.dev/oidc-callback` | +| Semaphore `provider_url` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` | +| Semaphore `redirect_url` | `https://semaphore.home.keithjackson.dev/api/...` | `https://semaphore.prod.home.keithjackson.dev/api/...` | + +Dev's values are byte-identical to what it runs today — this change moves prod only. + +### [parity] the apiserver invariant HOLDS + +Prod's kube-apiserver `oidc-issuer-url` is an **install-time k3s flag** derived from +`k3s_base_domain` (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` = `prod.home.keithjackson.dev`), +not from `cluster-vars` — it cannot read the Flux variable. Headlamp forwards the user's `id_token` +to the apiserver, so the two must name the same issuer. After this change they do, in both +environments, because both derive from the same per-environment domain. Unchanged risk, now written +down in `platform/headlamp/README.md`: moving one without the other 401s every request. + +### Also fixed for the next person + +`headlamp-oidc-credentials.sops.yaml.example` used to *teach* the broken pattern — it listed +`OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` as keys to fill in. It now says, in the file, that adding +either key back re-breaks prod.