fix(semaphore): per-environment OIDC URLs — same latent break as Headlamp

Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.

FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).

Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.

apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.

Co-Authored-By: Claude Opus 5 <[email protected]>

catalog/semaphore.yaml

@@ -78,16 +78,34 @@ env:
   #     (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
   #     JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
   #     client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
-  #     token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
-  #     semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
-  #     keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
-  #     kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
-  #     The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
-  - name: SEMAPHORE_OIDC_PROVIDERS
+  #     token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
+  #     SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
+  #
+  #     PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
+  #     semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
+  #     client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
+  #     redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
+  #     reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
+  #     prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
+  #     192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
+  #     platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
+  #
+  #     THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
+  #     ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
+  #     client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
+  #     in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
+  #     var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
+  #     SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
+  #     order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
+  #     fail-loud in the way that matters: if the Secret key is missing the Pod never starts
+  #     (CreateContainerConfigError), rather than silently authenticating against nothing. ---
+  - name: SEMAPHORE_OIDC_CLIENT_SECRET
     valueFrom:
       secretKeyRef:
         name: semaphore-bootstrap
-        key: oidc-providers-json
+        key: oidc-client-secret
+  - name: SEMAPHORE_OIDC_PROVIDERS
+    value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
 # The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
 # and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
 # is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
@@ -117,10 +135,12 @@ auth:
   groups:
     - platform-admins           # access scoped to an authorized group (R-IDP-06)
   # WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
-  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
-  # in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
-  # authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
-  # never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
+  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
+  # (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
+  # client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
+  # The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
+  # break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
+  # No secret is embedded here.
 update:
   channel: stable               # R-UPD-01
   range: ">=2.19.0 <3.0.0"      # stay on the 2.x stable line