Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1 renders a container arg the appVersion v0.40.1 backend lacks; upstream kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy, apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready". Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move all Headlamp resources into a new non-gating Flux Kustomization clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly. The new Kustomization owns the whole platform/headlamp component; its HTTPRoute moves from platform-config into platform/headlamp/kustomization.yaml. Removed headlamp from platform/kustomization.yaml and its httproute from platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall platform/platform-config/apps. Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that same image (appVersion == chart version), so chart+image realign; every arg the chart renders for our values is supported. OIDC config + per-user RBAC (clusterRoleBinding.create:false) unchanged. Gate: `make -C ci gate` PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)
Headlamp is served at https://headlamp.home.keithjackson.dev behind the shared Gateway
(wildcard TLS), with native Keycloak OIDC login and per-user RBAC (ADR-0014, docs/15).
How per-user RBAC works (end to end)
- A user logs in at Keycloak (
homelabrealm,headlampconfidential client, auth-code flow). - Headlamp's backend keeps the user's
id_tokenand forwards it as the bearer on every request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour). - The apiserver validates that token — it is configured (
infra/ansibleroles/k3s_server) withoidc-issuer-url= this realm,oidc-client-id=kubernetes,oidc-username-claim=email,oidc-groups-claim=groups, prefixesoidc:. - RBAC is applied per user via the existing group bindings in
../keycloak/k8s-oidc-rbac.yaml:oidc:platform-admins→cluster-admin,oidc:viewers→view.
So each user gets exactly their own rights — not a shared Headlamp identity. That is why the
Headlamp ServiceAccount is not bound to cluster-admin: the chart default
clusterRoleBinding.create: true (→ cluster-admin) is overridden to false in
helmrelease.yaml. User requests ride the forwarded user token; the SA needs no cluster rights.
The audience nuance (why the client has an audience mapper)
The apiserver only accepts ID tokens whose aud contains kubernetes (oidc-client-id= kubernetes). A plain headlamp client issues aud=[headlamp], which the apiserver would reject.
The headlamp client therefore carries an oidc-audience-mapper (see
../keycloak/keycloak-realm-import.yaml) with included.client.audience: kubernetes, so tokens
carry aud=[headlamp, kubernetes]:
- Headlamp's own verifier accepts it (aud contains
headlamp), and - the apiserver accepts it (aud contains
kubernetes).
This keeps the apiserver OIDC config untouched (preferred over widening --api-audiences).
Secrets
- The confidential client secret lives SOPS-encrypted in
../keycloak/keycloak-realm-clients.sops.yaml(headlamp-client-secret, injected into the realm import via theHEADLAMP_CLIENT_SECRETplaceholder) and mirrored in this namespace'sheadlamp-oidc-credentials.sops.yaml(clientSecret), which the chartconfig.oidc.externalSecretenvFrom's. Same cross-namespace mirror pattern as grafana/gitea.
Break-glass
The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full access if Keycloak/OIDC is down — the standard break-glass path.
Files
| File | Tier | Purpose |
|---|---|---|
namespace.yaml |
platform | headlamp namespace (gateway-access=shared; platform, not app-namespace) |
helmrepository.yaml / helmrelease.yaml |
platform | Official kubernetes-sigs/headlamp chart + OIDC/RBAC values |
headlamp-oidc-credentials.sops.yaml |
platform | SOPS OIDC config the chart envFrom's (decrypted by the platform Flux Kustomization) |
networkpolicy.yaml |
platform | default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows |
httproute.yaml |
platform-config | Gateway API HTTPRoute for headlamp.home.keithjackson.dev (needs Gateway CRDs) |