0 Fork 0

Files

T

jakman85andClaude Opus 5 bfa39bc5d0 fix(headlamp): per-environment OIDC URLs — prod was dialing DEV's Keycloak

The operator could not log in to Headlamp on prod:

  Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
  dial tcp 192.168.1.53:443: i/o timeout

ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted
headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites
the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen
at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked
correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak
at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24.

FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the
HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per
environment. The Secret keeps only what is genuinely secret or environment-invariant
(OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES).

Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit
env entry silently WINS over the same name from envFrom — so the two sets must not overlap,
and they no longer do. The five load-bearing names the chart's $(...) args require
(OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered;
verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and
`helm template` of chart 0.41.0 against the substituted values.

Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT
`-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via
`env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL
had been dead weight. The value is byte-identical to the single redirectUri registered on the
Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts.

[parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from
k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-07 02:06:38 -05:00

6.1 KiB

Raw Permalink Blame History

Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)

Headlamp is served at https://headlamp.${BASE_DOMAIN} behind the shared Gateway (wildcard TLS), with native Keycloak OIDC login and per-user RBAC (ADR-0014, docs/15).

How per-user RBAC works (end to end)

A user logs in at Keycloak (homelab realm, headlamp confidential client, auth-code flow).
Headlamp's backend keeps the user's id_token and forwards it as the bearer on every request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour).
The apiserver validates that token — it is configured (infra/ansible roles/k3s_server) with oidc-issuer-url = this realm, oidc-client-id=kubernetes, oidc-username-claim=email, oidc-groups-claim=groups, prefixes oidc:.
RBAC is applied per user via the existing group bindings in ../keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins → cluster-admin, oidc:viewers → view.

So each user gets exactly their own rights — not a shared Headlamp identity. That is why the Headlamp ServiceAccount is not bound to cluster-admin: the chart default clusterRoleBinding.create: true (→ cluster-admin) is overridden to false in helmrelease.yaml. User requests ride the forwarded user token; the SA needs no cluster rights.

The audience nuance (why the client has an audience mapper)

The apiserver only accepts ID tokens whose aud contains kubernetes (oidc-client-id= kubernetes). A plain headlamp client issues aud=[headlamp], which the apiserver would reject. The headlamp client therefore carries an oidc-audience-mapper (see ../keycloak/keycloak-realm-import.yaml) with included.client.audience: kubernetes, so tokens carry aud=[headlamp, kubernetes]:

Headlamp's own verifier accepts it (aud contains headlamp), and
the apiserver accepts it (aud contains kubernetes).

This keeps the apiserver OIDC config untouched (preferred over widening --api-audiences).

Where the OIDC config comes from (two sources, no overlap)

The container needs five env vars, and their names are load-bearing: the chart's args reference them via $(...) (-oidc-idp-issuer-url=$(OIDC_ISSUER_URL), -oidc-client-id=$(OIDC_CLIENT_ID), …). They must be exactly OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_ISSUER_URL, OIDC_SCOPES, OIDC_CALLBACK_URL — not camelCase. A misnamed or missing name does not fail loudly: Kubernetes leaves the literal string $(OIDC_ISSUER_URL) in the argument and login dies with unsupported protocol scheme "".

Env var	Source	Why
`OIDC_CLIENT_ID`, `OIDC_SCOPES`, `OIDC_CLIENT_SECRET`	`headlamp-oidc-credentials.sops.yaml`, `envFrom`'d by `config.oidc.externalSecret`	secret, or environment-invariant
`OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL`	`helmrelease.yaml` → `spec.values.env`, templated `${BASE_DOMAIN}`	environment-specific and public

Exactly one source per variable. Never define the same name in both places: an explicit env: entry silently overrides the same name arriving via envFrom, so an overlap hides which value is actually live.

Why the URLs are not in the Secret (the 2026-09-07 prod outage)

Flux's ${BASE_DOMAIN} postBuild substitution runs over the rendered manifest text; it can never reach inside an encrypted Secret. OIDC_ISSUER_URL and OIDC_CALLBACK_URL used to be keys in headlamp-oidc-credentials.sops.yaml, so both were frozen at the apex home.keithjackson.dev. Dev is the apex, so dev looked fine — but prod (BASE_DOMAIN = prod.home.keithjackson.dev) was pointed at dev's Keycloak, 192.168.1.53, which sits on the dev LAN behind pfSense and is unroutable from prod's 192.168.2.0/24. Every prod login failed with:

Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
dial tcp 192.168.1.53:443: i/o timeout

An issuer URL and a callback URL are public, so they now live in cleartext in helmrelease.yaml where substitution works. Only the client secret stays encrypted. The same root cause and the same fix apply to Semaphore (catalog/semaphore.yaml, SEMAPHORE_OIDC_PROVIDERS).

[parity] OIDC_ISSUER_URL must equal the kube-apiserver's oidc-issuer-url, which is an install-time k3s flag set from k3s_base_domain (infra/ansible/roles/k3s_server/defaults/main.yml, overridden per inventory group), not from cluster-vars. Headlamp forwards the user's id_token to the apiserver, so a mismatch 401s every request. Both derive from the same per-environment domain — dev home.keithjackson.dev, prod prod.home.keithjackson.dev — which is what keeps them in step; changing one without the other breaks login.

Secrets

The confidential client secret lives SOPS-encrypted in ../keycloak/keycloak-realm-clients.sops.yaml (headlamp-client-secret, injected into the realm import via the HEADLAMP_CLIENT_SECRET placeholder) and mirrored in this namespace's headlamp-oidc-credentials.sops.yaml (OIDC_CLIENT_SECRET), which the chart config.oidc.externalSecret envFrom's. Same cross-namespace mirror pattern as grafana/gitea.

Break-glass

The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full access if Keycloak/OIDC is down — the standard break-glass path.

Files

File	Tier	Purpose
`namespace.yaml`	platform	`headlamp` namespace (`gateway-access=shared`; platform, not app-namespace)
`helmrepository.yaml` / `helmrelease.yaml`	platform	Official `kubernetes-sigs/headlamp` chart + OIDC/RBAC values
`headlamp-oidc-credentials.sops.yaml`	platform	SOPS OIDC secret the chart envFrom's — client id/scopes/secret only; the per-env URLs live in `helmrelease.yaml` `spec.values.env`
`networkpolicy.yaml`	platform	default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows
`httproute.yaml`	platform-config	Gateway API HTTPRoute for `headlamp.${BASE_DOMAIN}` (needs Gateway CRDs)

6.1 KiB Raw Permalink Blame History