merge: OIDC issuer/callback URLs become per-environment
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

This commit is contained in:
jakman85 committed 2026-09-07 02:12:15 -05:00
commit 09265e0381
8 files changed
+266 -66

No files matched your search

+89
View File
@@ -1758,3 +1758,92 @@ into a transient hard Kustomization failure.
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
## 2026-09-07 — LIVE BUG: prod's OIDC clients were pointed at DEV's Keycloak. Secrets are a substitution blind spot.
The operator could not log in to Headlamp on prod:
```
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
dial tcp 192.168.1.53:443: i/o timeout
```
`keycloak.home.keithjackson.dev` is **dev's** Keycloak. It resolves to `192.168.1.53`, on the dev
LAN behind pfSense; prod lives on `192.168.2.0/24` (pfSense's WAN) and cannot route there, so the
discovery fetch simply hung until it timed out.
### Root cause — the class, not the instance
**Flux's `${BASE_DOMAIN}` postBuild substitution rewrites the RENDERED MANIFEST TEXT. It can never
reach inside a SOPS-encrypted Secret.** Environment-specific URLs had been stored *inside* two
encrypted Secrets purely because it was convenient to keep one blob per app:
| File | Buried, un-substitutable values |
|---|---|
| `platform/headlamp/headlamp-oidc-credentials.sops.yaml` | `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` |
| `apps/wave-1/semaphore-bootstrap-secret.sops.yaml` (`oidc-providers-json`) | the realm issuer + the OIDC redirect URI |
Both were frozen at the apex `home.keithjackson.dev`. **Dev IS the apex, so dev was correct by
coincidence** — the values were never per-environment, they just happened to match one environment.
Prod was wrong from the first reconcile. Semaphore on prod was broken the same way and nobody had
noticed, because nobody had tried prod SSO yet.
This is the same failure shape as the 2026-09-01 nextcloud guard: **a value that silently resolves
to something plausible is worse than one that fails.** `StrictPostBuildSubstitutions` cannot catch
it either — there is no `${...}` left unresolved to complain about; the ciphertext never contained
one.
### The rule this establishes
> **Non-secret, environment-specific values MUST NOT live inside an encrypted Secret.** Issuer URLs,
> callback URLs, redirect URIs and hostnames are public. Encrypt the secret half only, and template
> the rest where `${BASE_DOMAIN}` can reach it.
### The fix
- **Headlamp** — `OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` moved to the HelmRelease's
`spec.values.env`, templated `${BASE_DOMAIN}`. The Secret keeps `OIDC_CLIENT_SECRET` +
the environment-invariant `OIDC_CLIENT_ID` / `OIDC_SCOPES`, still `envFrom`'d.
**Exactly one source per variable** — an explicit `env:` entry silently outranks the same name
arriving via `envFrom`, so the two sets deliberately do not overlap.
Side effect, intended: naming `OIDC_CALLBACK_URL` in `env:` makes chart 0.41.0 finally emit
`-oidc-callback-url=$(OIDC_CALLBACK_URL)`; it gates that arg on `env:`/`config.oidc`, which a
Secret key alone never tripped — so the Secret's callback URL had been dead weight all along.
- **Semaphore** — the `SEMAPHORE_OIDC_PROVIDERS` JSON is now a plaintext literal in
`catalog/semaphore.yaml` (re-rendered to the `apps/wave-1` golden). Only the client secret stays
encrypted, in its own key `oidc-client-secret`, and the **kubelet** splices it back via
dependent-env-var expansion: `$(SEMAPHORE_OIDC_CLIENT_SECRET)` resolves against the `secretKeyRef`
env var declared immediately above it. **Order is load-bearing** — the kubelet resolves `env` in
list order and leaves a forward reference as literal text. `flux envsubst` ignores `$(...)`, so
the two expansion layers do not collide (the `$${VAR}` escape in `apps/wave-1/nextcloud.yaml`
documents the one form that DOES need escaping).
### Proof
Rendered per overlay (`kubectl kustomize --load-restrictor LoadRestrictionsNone`, then
`flux envsubst --strict` with that overlay's `cluster-vars`), plus `helm template` of headlamp
0.41.0 against the substituted values:
| | dev | prod |
|---|---|---|
| Headlamp `OIDC_ISSUER_URL` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` |
| Headlamp `OIDC_CALLBACK_URL` | `https://headlamp.home.keithjackson.dev/oidc-callback` | `https://headlamp.prod.home.keithjackson.dev/oidc-callback` |
| Semaphore `provider_url` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` |
| Semaphore `redirect_url` | `https://semaphore.home.keithjackson.dev/api/...` | `https://semaphore.prod.home.keithjackson.dev/api/...` |
Dev's values are byte-identical to what it runs today — this change moves prod only.
### [parity] the apiserver invariant HOLDS
Prod's kube-apiserver `oidc-issuer-url` is an **install-time k3s flag** derived from
`k3s_base_domain` (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` = `prod.home.keithjackson.dev`),
not from `cluster-vars` — it cannot read the Flux variable. Headlamp forwards the user's `id_token`
to the apiserver, so the two must name the same issuer. After this change they do, in both
environments, because both derive from the same per-environment domain. Unchanged risk, now written
down in `platform/headlamp/README.md`: moving one without the other 401s every request.
### Also fixed for the next person
`headlamp-oidc-credentials.sops.yaml.example` used to *teach* the broken pattern — it listed
`OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` as keys to fill in. It now says, in the file, that adding
either key back re-breaks prod.
@@ -5,6 +5,15 @@
# credential from its own namespace (a Pod cannot read a Secret in another namespace).
# access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
# credentials; MUST stay stable across restarts or stored secrets become undecryptable.
# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
# `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
# injects that copy into the client), same cross-namespace mirror pattern as db-password above.
# It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
# SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
# URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
# so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
# catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
# back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
# Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
apiVersion: v1
kind: Secret
@@ -15,11 +24,11 @@ metadata:
app.kubernetes.io/name: semaphore
type: Opaque
stringData:
admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
sops:
kms: []
gcp_kms: []
@@ -29,14 +38,14 @@ sops:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-08-27T22:12:32Z"
mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
lastmodified: "2026-09-07T07:01:50Z"
mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+4 -2
View File
@@ -118,11 +118,13 @@ spec:
key: access-key-encryption
- name: SEMAPHORE_PORT
value: :3000
- name: SEMAPHORE_OIDC_PROVIDERS
- name: SEMAPHORE_OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: semaphore-bootstrap
key: oidc-providers-json
key: oidc-client-secret
- name: SEMAPHORE_OIDC_PROVIDERS
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
volumeMounts:
- name: config
mountPath: /etc/semaphore
+31 -11
View File
@@ -78,16 +78,34 @@ env:
# (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
# JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
# client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
# token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
# semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
# keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
# kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
# The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
- name: SEMAPHORE_OIDC_PROVIDERS
# token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
# SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
#
# PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
# semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
# client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
# redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
# reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
# prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
# 192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
# platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
#
# THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
# ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
# client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
# in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
# var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
# SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
# order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
# fail-loud in the way that matters: if the Secret key is missing the Pod never starts
# (CreateContainerConfigError), rather than silently authenticating against nothing. ---
- name: SEMAPHORE_OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: semaphore-bootstrap
key: oidc-providers-json
key: oidc-client-secret
- name: SEMAPHORE_OIDC_PROVIDERS
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
# The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
# and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
# is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
@@ -117,10 +135,12 @@ auth:
groups:
- platform-admins # access scoped to an authorized group (R-IDP-06)
# WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
# in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
# authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
# never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
# (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
# client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
# The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
# break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
# No secret is embedded here.
update:
channel: stable # R-UPD-01
range: ">=2.19.0 <3.0.0" # stay on the 2.x stable line
+46 -8
View File
@@ -1,6 +1,6 @@
# Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)
Headlamp is served at `https://headlamp.home.keithjackson.dev` behind the shared Gateway
Headlamp is served at `https://headlamp.${BASE_DOMAIN}` behind the shared Gateway
(wildcard TLS), with **native Keycloak OIDC login** and **per-user RBAC** (ADR-0014, docs/15).
## How per-user RBAC works (end to end)
@@ -33,6 +33,49 @@ carry `aud=[headlamp, kubernetes]`:
This keeps the apiserver OIDC config untouched (preferred over widening `--api-audiences`).
## Where the OIDC config comes from (two sources, no overlap)
The container needs five env vars, and their **names are load-bearing**: the chart's args reference
them via `$(...)` (`-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-client-id=$(OIDC_CLIENT_ID)`, …).
They must be exactly `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_ISSUER_URL`, `OIDC_SCOPES`,
`OIDC_CALLBACK_URL` — **not** camelCase. A misnamed or missing name does **not** fail loudly:
Kubernetes leaves the literal string `$(OIDC_ISSUER_URL)` in the argument and login dies with
`unsupported protocol scheme ""`.
| Env var | Source | Why |
|---|---|---|
| `OIDC_CLIENT_ID`, `OIDC_SCOPES`, `OIDC_CLIENT_SECRET` | `headlamp-oidc-credentials.sops.yaml`, `envFrom`'d by `config.oidc.externalSecret` | secret, or environment-invariant |
| `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` | `helmrelease.yaml` → `spec.values.env`, templated `${BASE_DOMAIN}` | **environment-specific and public** |
**Exactly one source per variable.** Never define the same name in both places: an explicit `env:`
entry silently overrides the same name arriving via `envFrom`, so an overlap hides which value is
actually live.
### Why the URLs are not in the Secret (the 2026-09-07 prod outage)
Flux's `${BASE_DOMAIN}` postBuild substitution runs over the **rendered manifest text**; it can never
reach inside an encrypted Secret. `OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` used to be keys in
`headlamp-oidc-credentials.sops.yaml`, so both were frozen at the apex `home.keithjackson.dev`.
Dev **is** the apex, so dev looked fine — but prod (`BASE_DOMAIN = prod.home.keithjackson.dev`) was
pointed at **dev's** Keycloak, `192.168.1.53`, which sits on the dev LAN behind pfSense and is
unroutable from prod's `192.168.2.0/24`. Every prod login failed with:
```
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
dial tcp 192.168.1.53:443: i/o timeout
```
An issuer URL and a callback URL are public, so they now live in cleartext in `helmrelease.yaml`
where substitution works. Only the client secret stays encrypted. The same root cause and the same
fix apply to Semaphore (`catalog/semaphore.yaml`, `SEMAPHORE_OIDC_PROVIDERS`).
**[parity] `OIDC_ISSUER_URL` must equal the kube-apiserver's `oidc-issuer-url`**, which is an
install-time k3s flag set from `k3s_base_domain` (`infra/ansible/roles/k3s_server/defaults/main.yml`,
overridden per inventory group), *not* from `cluster-vars`. Headlamp forwards the user's `id_token`
to the apiserver, so a mismatch 401s every request. Both derive from the same per-environment
domain — dev `home.keithjackson.dev`, prod `prod.home.keithjackson.dev` — which is what keeps them
in step; changing one without the other breaks login.
## Secrets
- The confidential client secret lives SOPS-encrypted in
@@ -41,11 +84,6 @@ This keeps the apiserver OIDC config untouched (preferred over widening `--api-a
placeholder) and mirrored in this namespace's `headlamp-oidc-credentials.sops.yaml`
(`OIDC_CLIENT_SECRET`), which the chart `config.oidc.externalSecret` envFrom's. Same cross-namespace
mirror pattern as grafana/gitea.
- **Key names are load-bearing.** The chart `envFrom`s this Secret verbatim, so its keys become the
container env vars referenced by the chart's OIDC args (`-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`,
etc.). They must be exactly `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_ISSUER_URL`, `OIDC_SCOPES`
(and `OIDC_CALLBACK_URL`) — **not** camelCase. A misnamed key leaves `$(OIDC_ISSUER_URL)`
unexpanded and OIDC login fails with `unsupported protocol scheme ""`.
## Break-glass
@@ -58,6 +96,6 @@ access if Keycloak/OIDC is down — the standard break-glass path.
|---|---|---|
| `namespace.yaml` | platform | `headlamp` namespace (`gateway-access=shared`; platform, not app-namespace) |
| `helmrepository.yaml` / `helmrelease.yaml` | platform | Official `kubernetes-sigs/headlamp` chart + OIDC/RBAC values |
| `headlamp-oidc-credentials.sops.yaml` | platform | SOPS OIDC config the chart envFrom's (decrypted by the `platform` Flux Kustomization) |
| `headlamp-oidc-credentials.sops.yaml` | platform | SOPS OIDC **secret** the chart envFrom's — client id/scopes/secret only; the per-env URLs live in `helmrelease.yaml` `spec.values.env` |
| `networkpolicy.yaml` | platform | default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows |
| `httproute.yaml` | platform-config | Gateway API HTTPRoute for `headlamp.home.keithjackson.dev` (needs Gateway CRDs) |
| `httproute.yaml` | platform-config | Gateway API HTTPRoute for `headlamp.${BASE_DOMAIN}` (needs Gateway CRDs) |
@@ -5,11 +5,9 @@ metadata:
namespace: headlamp
type: Opaque
stringData:
OIDC_CLIENT_ID: ENC[AES256_GCM,data:dXAzoFfar4A=,iv:dHOJFcS6tpvtJes1c++nU6BbHyRlmGuLpncucbkJ4xA=,tag:lFI+KJPid2f/2r/ZjYlFTg==,type:str]
OIDC_ISSUER_URL: ENC[AES256_GCM,data:Er1ljhH62WahtEJkNvxk1BA2jz28OOCZmmyjkbVN82e/TFM7O+FB/FPjWUaW3lhd+dK16ro=,iv:Z3aP7+T+ytItYpbxjoYymLNGa7m8KGic4WqIQrrBSGE=,tag:/Rvhp5+r7AXIj2uDgL/dPg==,type:str]
OIDC_SCOPES: ENC[AES256_GCM,data:WoGaQKY9BFsi5/kDlRMXatINtmE=,iv:kTN+C176I8o5ftzZiKR10s9gl9VvBsOCJhFIZzj9gm4=,tag:ZyRIykD1h5YBq4grd6FB/w==,type:str]
OIDC_CALLBACK_URL: ENC[AES256_GCM,data:SZdpxEuW1BJccLyRGjuIDWBu6KpySARv/6M0mWtzc9jS+v64Lgr4VWKwgW0tdZrH5jM9rw==,iv:UbLwgfz7J0SIWbcQ4csqxv5B3enTmfHJzSDw/lmqAEI=,tag:lge2AuoQXK/u3uZbWvGnLQ==,type:str]
OIDC_CLIENT_SECRET: ENC[AES256_GCM,data:r0pb73gat8E7BFLPbdhguLh8G5PUT7PlJJeIcv7QPdMnN4iIB8MedMwxd77DkbxHSR7bav2ohQczjnbJBA4tPA==,iv:O0RbMFz4UD1sy8mbY7SXWCSGNF+lapDTL7tpphEcx7c=,tag:bEyuEs/lsK28fJeYDpYiOg==,type:str]
OIDC_CLIENT_ID: ENC[AES256_GCM,data:FCW6yV5oUCM=,iv:+oWwATPxrukg7BbNDraaWhBoLmRsAAlF/HwlyYtLBgI=,tag:aoEMlyxyXLeiLak+i3Suww==,type:str]
OIDC_SCOPES: ENC[AES256_GCM,data:k9jHaTSEQ7zl0gmK2zTW7AAbQ8c=,iv:cOGN1Oi/E9tTSZqNKY6i35wWk6fXatlYNRRhtVQrl1E=,tag:Q3I2pJ/a6ESX/ox2VvNDiA==,type:str]
OIDC_CLIENT_SECRET: ENC[AES256_GCM,data:qoWJYdYx5bZztHJRYlGgxVRy/n9izkgffoGuQwePWtV6YrQpxZGgS3sak8cP6tt5M+3ZkFSrysxgcqcjrzJFuA==,iv:ZtEfWbPOucVnYv3x107XEjZ8Qt7HXBpwTlSuoxv+x5I=,tag:PXuEyhIoPOxo/Aw6hEhfKw==,type:str]
sops:
kms: []
gcp_kms: []
@@ -19,14 +17,14 @@ sops:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1UDV1RzRWeGZvdWxFQlgz
dCtjTU8wZzR5Tks5dmYreFliREZGaDdXMVVjCkVPY2lZby9LNzJMelJkTlE1Yks2
cXN5YmhlaUtwN3ZjQUo5R0xnQWNQYm8KLS0tIDBMSUMxZ1pHaDRnUWs5RGtRNlhI
N0I4ekhqMWVWUFZoeGU4SERpYXVDQUkKUUwY6ZxEiWGdqV8s3wbnbPF9SkZJoxfy
3MkNQ7rZHSILFTPVaZSErNTI21vqQr3lG24+yKLfNhAZBNyB5lvedQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFdDZ6Zm9hSW9NY1Mrd0FO
NXQxdC9NWXlrd2VXUjBYeGVLREJjOU56dmdJCk15NzlCdFRaWDdvUWVTcWtRc3hj
eVJCNVdYSFZQU0V4Q0txTzhEMHVJMUEKLS0tIG5kamFsU3RwTWp6MWRlNTlQRlE3
MFJFZ2NvUXAvc3J6NUl2b0NHNUVDYTQKb7tibY/OHM4FRfQ2dzTnIRTIgWG8gddU
lFJCZG03U79MEG/UeottO4uNJ3jSXImXpysXSHnH3o5muNvIuEN9rw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-08-29T20:04:33Z"
mac: ENC[AES256_GCM,data:Haj8wMqDuIdtBopP02L1Z63+T8f0fcuSDpF5jnjgCB1+uzX+d4ylqpcVP5r8Ug8M/2ay2nYWoJBWzL0Y1iHcnDP7/T60M8hT46cBYPrDh3MqY0rWF5GwpjJxL9tStoSlDTYICeilAPqo2Vw4UM1CqnZv+th40rz5/R4MwRsU04w=,iv:5HPvtnF6nnDYDaT84r1JWf+ZH1X5/L1Aiim2lgOucH4=,tag:VM/2FXNW7sdjCk/k2AVl+g==,type:str]
lastmodified: "2026-09-07T07:04:07Z"
mac: ENC[AES256_GCM,data:hGJmhdXg0XaaCh6YRI7Wj1TlnjFtckASY4OSyfEgLM8tbwPoElXLir5HQJz7F19uPirbSslVuG8+s58Zvv4/JG/Qki9tQyTItJzbkFVg7iXnXOcYwxW8vw+P26NeE84CYXCIGGunzKZuY7/6ftul8czt/3p83I0X4Y4YzbttaGs=,iv:i2wY7tPaeizv1k7O7b9bMSdbGABFDri5WdDsOxGBkOw=,tag:QI1b2rYgeme26NoGBm3GEA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
@@ -2,15 +2,27 @@
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
#
# WHAT THIS IS: the OIDC config Headlamp uses to authenticate against the Keycloak `homelab` realm's
# WHAT THIS IS: the SECRET half of Headlamp's OIDC config for the Keycloak `homelab` realm's
# `headlamp` client. The Headlamp chart's `config.oidc.externalSecret` (helmrelease.yaml) envFrom's
# THIS Secret, so ALL OIDC env (OIDC_CLIENT_ID/OIDC_CLIENT_SECRET/OIDC_ISSUER_URL/OIDC_SCOPES/OIDC_CALLBACK_URL) live here,
# SOPS-encrypted, referenced by name only (R-IDP-04). The chart `envFrom`s this Secret VERBATIM, so
# the KEYS ARE the container env var names — they MUST be exactly OIDC_CLIENT_ID / OIDC_CLIENT_SECRET
# / OIDC_ISSUER_URL / OIDC_SCOPES / OIDC_CALLBACK_URL (the chart's `-oidc-*` args reference those
# names via $(...)). There is NO camelCase→OIDC_ mapping; a mismatched key breaks OIDC login.
# THIS Secret, so its KEYS ARE the container env var names — the chart renames nothing, and its
# `-oidc-*` args reference those names via $(...). They MUST be spelled exactly as below.
#
# The `clientSecret` value MUST equal the `headlamp-client-secret` placed in
# ── THE URLS ARE NOT HERE, AND MUST NOT BE PUT BACK (2026-09-07) ─────────────────────────────
# OIDC_ISSUER_URL and OIDC_CALLBACK_URL are ENVIRONMENT-SPECIFIC and NON-SECRET. They live in
# `helmrelease.yaml` under `spec.values.env`, written as
# https://keycloak.${BASE_DOMAIN}/realms/homelab
# https://headlamp.${BASE_DOMAIN}/oidc-callback
# so Flux's postBuild `${BASE_DOMAIN}` substitution resolves them per environment. They used to be
# keys in this Secret; substitution CANNOT reach inside ciphertext, so both were frozen at the apex
# `home.keithjackson.dev` and PROD pointed Headlamp at DEV's Keycloak (192.168.1.53 — on the dev LAN
# behind pfSense, unroutable from prod's 192.168.2.0/24). Every prod login failed with
# dial tcp 192.168.1.53:443: i/o timeout on .../.well-known/openid-configuration
# Dev only ever looked correct because dev IS the apex. Adding either key back here re-breaks prod.
#
# ONE SOURCE PER VARIABLE: never define a name in BOTH this Secret and `spec.values.env` — an
# explicit `env:` entry silently WINS over the same name from `envFrom`, hiding which one is live.
#
# The `OIDC_CLIENT_SECRET` value MUST equal the `headlamp-client-secret` placed in
# platform/keycloak/keycloak-realm-clients.sops.yaml (which the realm import injects into the
# Keycloak `headlamp` client) — same cross-namespace mirror pattern as grafana/gitea (a Secret
# cannot be shared across namespaces, so the one value is stored in two encrypted files).
@@ -18,7 +30,7 @@
# HAND-OFF STEPS (operator):
# (a) Use the SAME headlamp-client-secret generated for keycloak-realm-clients (openssl rand -hex 32).
# (b) Copy to: platform/headlamp/headlamp-oidc-credentials.sops.yaml
# (c) Fill `clientSecret`; encrypt:
# (c) Fill `OIDC_CLIENT_SECRET`; encrypt:
# sops -e -i platform/headlamp/headlamp-oidc-credentials.sops.yaml
# (d) It is already listed under `resources:` in platform/headlamp/kustomization.yaml.
# =============================================================================
@@ -32,14 +44,12 @@ stringData:
# KEY NAMES ARE LOAD-BEARING. The chart does NOT rename these — it `envFrom`s this Secret
# verbatim, so each key becomes a container env var of the SAME name, and the chart's OIDC args
# reference them by exact name: `-oidc-client-id=$(OIDC_CLIENT_ID)`,
# `-oidc-client-secret=$(OIDC_CLIENT_SECRET)`, `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`,
# `-oidc-scopes=$(OIDC_SCOPES)`. If a key is misnamed (e.g. camelCase `issuerURL`), the env var
# `OIDC_ISSUER_URL` is undefined, K8s leaves the literal string `$(OIDC_ISSUER_URL)` in the arg,
# and Headlamp's OIDC discovery fails with `unsupported protocol scheme ""`. Use these EXACT keys.
# Non-secret OIDC config (kept in the encrypted Secret because the chart envFrom's a single Secret).
# `-oidc-client-secret=$(OIDC_CLIENT_SECRET)`, `-oidc-scopes=$(OIDC_SCOPES)`. If a key is misnamed
# (e.g. camelCase `clientID`), the env var is undefined, K8s leaves the LITERAL string
# `$(OIDC_CLIENT_ID)` in the arg — it does NOT fail loudly — and OIDC login breaks. Use these
# EXACT keys. Environment-INVARIANT, non-secret values stay here simply because the chart wants a
# single envFrom source; anything that varies per environment belongs in helmrelease.yaml `env:`.
OIDC_CLIENT_ID: headlamp
OIDC_ISSUER_URL: https://keycloak.home.keithjackson.dev/realms/homelab
OIDC_SCOPES: "openid email profile"
OIDC_CALLBACK_URL: https://headlamp.home.keithjackson.dev/oidc-callback
# The confidential client secret — MUST match keycloak-realm-clients headlamp-client-secret.
OIDC_CLIENT_SECRET: REPLACE-ME
+40 -6
View File
@@ -58,16 +58,50 @@ spec:
config:
inCluster: true
oidc:
# OIDC config (OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES /
# OIDC_CALLBACK_URL) is supplied via an EXISTING SOPS-encrypted Secret consumed by name only
# (R-IDP-04) — the chart envFrom's it. The Secret keys ARE the env var names (no renaming),
# and the chart's `-oidc-*` args reference them via $(...), e.g.
# `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`; a camelCase key would leave that arg unexpanded
# and break OIDC login (`unsupported protocol scheme ""`). Nothing sensitive is rendered here.
# OIDC config reaches the container as env vars whose NAMES ARE LOAD-BEARING: the chart's
# args reference them via $(...) — `-oidc-client-id=$(OIDC_CLIENT_ID)`,
# `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-callback-url=$(OIDC_CALLBACK_URL)`, … — so
# they MUST be exactly OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES /
# OIDC_CALLBACK_URL. A misnamed or absent name does NOT fail: Kubernetes leaves the LITERAL
# string `$(OIDC_ISSUER_URL)` in the arg and login dies with `unsupported protocol scheme ""`.
#
# There are exactly TWO sources, with NO variable served by both (a name defined in `env:`
# would silently WIN over the same name coming from `envFrom`, so overlap is forbidden here):
# • envFrom this SOPS Secret → OIDC_CLIENT_ID, OIDC_SCOPES, OIDC_CLIENT_SECRET
# • the `env:` block below → OIDC_ISSUER_URL, OIDC_CALLBACK_URL
# The Secret is consumed by name only (R-IDP-04); nothing sensitive is rendered here.
externalSecret:
enabled: true
name: headlamp-oidc
hasScopes: true
# ── PER-ENVIRONMENT OIDC URLs (2026-09-07 fix) ───────────────────────────────────────────────
# These two used to be keys INSIDE headlamp-oidc-credentials.sops.yaml. That was the bug: Flux's
# `${BASE_DOMAIN}` postBuild substitution operates on the RENDERED manifest text and can never
# reach inside an encrypted Secret, so both were frozen at the apex `home.keithjackson.dev`.
# Dev is the apex, so dev looked fine; PROD (BASE_DOMAIN = prod.home.keithjackson.dev) therefore
# pointed Headlamp at DEV's Keycloak — 192.168.1.53, on the dev LAN behind pfSense, unroutable
# from prod's 192.168.2.0/24 — and every login died with
# dial tcp 192.168.1.53:443: i/o timeout on .../.well-known/openid-configuration
# An issuer URL and a callback URL are PUBLIC. They belong in cleartext, where substitution works.
#
# [parity] OIDC_ISSUER_URL MUST equal the kube-apiserver's `oidc-issuer-url`, which is an
# INSTALL-TIME k3s flag derived from `k3s_base_domain` (infra/ansible roles/k3s_server/defaults +
# inventory/group_vars/k3s_<env>/vars.yml), NOT from this ConfigMap. Headlamp forwards the user's
# id_token to the apiserver as the bearer, so an issuer mismatch = every request 401s. Both sides
# are keyed off the same per-environment domain (BASE_DOMAIN == k3s_base_domain), which is what
# keeps them in step — dev home.keithjackson.dev, prod prod.home.keithjackson.dev.
#
# NOTE: naming OIDC_CALLBACK_URL here also makes the chart EMIT `-oidc-callback-url=$(…)`, which
# it previously omitted (the arg is gated on the value being set via `env:`/`config.oidc`, and the
# Secret key alone did not trip that gate — so the Secret's callback URL was dead weight). The
# value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client
# (platform/keycloak/keycloak-realm-import.yaml, also `${BASE_DOMAIN}`-templated), which is the
# only URI Keycloak will accept, so making it explicit cannot change behaviour in either env.
env:
- name: OIDC_ISSUER_URL
value: https://keycloak.${BASE_DOMAIN}/realms/homelab
- name: OIDC_CALLBACK_URL
value: https://headlamp.${BASE_DOMAIN}/oidc-callback
# PER-USER RBAC: do NOT grant Headlamp's ServiceAccount cluster-admin. User requests are
# authorized by their forwarded OIDC token against the existing oidc:* group bindings.
serviceAccount: