Files
home-network-k3s-impl/platform/headlamp
jakman85andClaude Opus 4.8 ebe0371b1b platform: move Headlamp out of the gating platform tier + fix -session-ttl crashloop
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".

Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.

Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.

Gate: `make -C ci gate` PASS.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 12:01:01 -05:00
..

Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)

Headlamp is served at https://headlamp.home.keithjackson.dev behind the shared Gateway (wildcard TLS), with native Keycloak OIDC login and per-user RBAC (ADR-0014, docs/15).

How per-user RBAC works (end to end)

  1. A user logs in at Keycloak (homelab realm, headlamp confidential client, auth-code flow).
  2. Headlamp's backend keeps the user's id_token and forwards it as the bearer on every request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour).
  3. The apiserver validates that token — it is configured (infra/ansible roles/k3s_server) with oidc-issuer-url = this realm, oidc-client-id=kubernetes, oidc-username-claim=email, oidc-groups-claim=groups, prefixes oidc:.
  4. RBAC is applied per user via the existing group bindings in ../keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins → cluster-admin, oidc:viewers → view.

So each user gets exactly their own rights — not a shared Headlamp identity. That is why the Headlamp ServiceAccount is not bound to cluster-admin: the chart default clusterRoleBinding.create: true (→ cluster-admin) is overridden to false in helmrelease.yaml. User requests ride the forwarded user token; the SA needs no cluster rights.

The audience nuance (why the client has an audience mapper)

The apiserver only accepts ID tokens whose aud contains kubernetes (oidc-client-id= kubernetes). A plain headlamp client issues aud=[headlamp], which the apiserver would reject. The headlamp client therefore carries an oidc-audience-mapper (see ../keycloak/keycloak-realm-import.yaml) with included.client.audience: kubernetes, so tokens carry aud=[headlamp, kubernetes]:

  • Headlamp's own verifier accepts it (aud contains headlamp), and
  • the apiserver accepts it (aud contains kubernetes).

This keeps the apiserver OIDC config untouched (preferred over widening --api-audiences).

Secrets

  • The confidential client secret lives SOPS-encrypted in ../keycloak/keycloak-realm-clients.sops.yaml (headlamp-client-secret, injected into the realm import via the HEADLAMP_CLIENT_SECRET placeholder) and mirrored in this namespace's headlamp-oidc-credentials.sops.yaml (clientSecret), which the chart config.oidc.externalSecret envFrom's. Same cross-namespace mirror pattern as grafana/gitea.

Break-glass

The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full access if Keycloak/OIDC is down — the standard break-glass path.

Files

File Tier Purpose
namespace.yaml platform headlamp namespace (gateway-access=shared; platform, not app-namespace)
helmrepository.yaml / helmrelease.yaml platform Official kubernetes-sigs/headlamp chart + OIDC/RBAC values
headlamp-oidc-credentials.sops.yaml platform SOPS OIDC config the chart envFrom's (decrypted by the platform Flux Kustomization)
networkpolicy.yaml platform default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows
httproute.yaml platform-config Gateway API HTTPRoute for headlamp.home.keithjackson.dev (needs Gateway CRDs)