home-network-k3s-impl/platform/headlamp

Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)

Headlamp is served at https://headlamp.${BASE_DOMAIN} behind the shared Gateway (wildcard TLS), with native Keycloak OIDC login and per-user RBAC (ADR-0014, docs/15).

How per-user RBAC works (end to end)

1. A user logs in at Keycloak (homelab realm, headlamp confidential client, auth-code flow).
2. Headlamp's backend keeps the user's id_token and forwards it as the bearer on every request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour).
3. The apiserver validates that token — it is configured (infra/ansible roles/k3s_server) with oidc-issuer-url = this realm, oidc-client-id=kubernetes, oidc-username-claim=email, oidc-groups-claim=groups, prefixes oidc:.
4. RBAC is applied per user via the existing group bindings in ../keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins → cluster-admin, oidc:viewers → view.

So each user gets exactly their own rights — not a shared Headlamp identity. That is why the Headlamp ServiceAccount is not bound to cluster-admin: the chart default clusterRoleBinding.create: true (→ cluster-admin) is overridden to false in helmrelease.yaml. User requests ride the forwarded user token; the SA needs no cluster rights.

The audience nuance (why the client has an audience mapper)

The apiserver only accepts ID tokens whose aud contains kubernetes (oidc-client-id= kubernetes). A plain headlamp client issues aud=[headlamp], which the apiserver would reject. The headlamp client therefore carries an oidc-audience-mapper (see ../keycloak/keycloak-realm-import.yaml) with included.client.audience: kubernetes, so tokens carry aud=[headlamp, kubernetes]:

• Headlamp's own verifier accepts it (aud contains headlamp), and
• the apiserver accepts it (aud contains kubernetes).

This keeps the apiserver OIDC config untouched (preferred over widening --api-audiences).

Where the OIDC config comes from (two sources, no overlap)

The container needs five env vars, and their names are load-bearing: the chart's args reference them via $(...) (-oidc-idp-issuer-url=$(OIDC_ISSUER_URL), -oidc-client-id=$(OIDC_CLIENT_ID), …). They must be exactly OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_ISSUER_URL, OIDC_SCOPES, OIDC_CALLBACK_URL — not camelCase. A misnamed or missing name does not fail loudly: Kubernetes leaves the literal string $(OIDC_ISSUER_URL) in the argument and login dies with unsupported protocol scheme "".

Env var	Source	Why
OIDC_CLIENT_ID, OIDC_SCOPES, OIDC_CLIENT_SECRET	headlamp-oidc-credentials.sops.yaml, envFrom'd by config.oidc.externalSecret	secret, or environment-invariant
OIDC_ISSUER_URL, OIDC_CALLBACK_URL	helmrelease.yaml → spec.values.env, templated ${BASE_DOMAIN}	environment-specific and public

Exactly one source per variable. Never define the same name in both places: an explicit env: entry silently overrides the same name arriving via envFrom, so an overlap hides which value is actually live.

Why the URLs are not in the Secret (the 2026-09-07 prod outage)

Flux's ${BASE_DOMAIN} postBuild substitution runs over the rendered manifest text; it can never reach inside an encrypted Secret. OIDC_ISSUER_URL and OIDC_CALLBACK_URL used to be keys in headlamp-oidc-credentials.sops.yaml, so both were frozen at the apex home.keithjackson.dev. Dev is the apex, so dev looked fine — but prod (BASE_DOMAIN = prod.home.keithjackson.dev) was pointed at dev's Keycloak, 192.168.1.53, which sits on the dev LAN behind pfSense and is unroutable from prod's 192.168.2.0/24. Every prod login failed with:

Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
dial tcp 192.168.1.53:443: i/o timeout

An issuer URL and a callback URL are public, so they now live in cleartext in helmrelease.yaml where substitution works. Only the client secret stays encrypted. The same root cause and the same fix apply to Semaphore (catalog/semaphore.yaml, SEMAPHORE_OIDC_PROVIDERS).

[parity] OIDC_ISSUER_URL must equal the kube-apiserver's oidc-issuer-url, which is an install-time k3s flag set from k3s_base_domain (infra/ansible/roles/k3s_server/defaults/main.yml, overridden per inventory group), not from cluster-vars. Headlamp forwards the user's id_token to the apiserver, so a mismatch 401s every request. Both derive from the same per-environment domain — dev home.keithjackson.dev, prod prod.home.keithjackson.dev — which is what keeps them in step; changing one without the other breaks login.

Secrets

• The confidential client secret lives SOPS-encrypted in ../keycloak/keycloak-realm-clients.sops.yaml (headlamp-client-secret, injected into the realm import via the HEADLAMP_CLIENT_SECRET placeholder) and mirrored in this namespace's headlamp-oidc-credentials.sops.yaml (OIDC_CLIENT_SECRET), which the chart config.oidc.externalSecret envFrom's. Same cross-namespace mirror pattern as grafana/gitea.

Break-glass

The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full access if Keycloak/OIDC is down — the standard break-glass path.

Files

File	Tier	Purpose
namespace.yaml	platform	headlamp namespace (gateway-access=shared; platform, not app-namespace)
helmrepository.yaml / helmrelease.yaml	platform	Official kubernetes-sigs/headlamp chart + OIDC/RBAC values
headlamp-oidc-credentials.sops.yaml	platform	SOPS OIDC secret the chart envFrom's — client id/scopes/secret only; the per-env URLs live in helmrelease.yaml spec.values.env
networkpolicy.yaml	platform	default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows
httproute.yaml	platform-config	Gateway API HTTPRoute for headlamp.${BASE_DOMAIN} (needs Gateway CRDs)