# Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)

Headlamp is served at `https://headlamp.${BASE_DOMAIN}` behind the shared Gateway
(wildcard TLS), with **native Keycloak OIDC login** and **per-user RBAC** (ADR-0014, docs/15).

## How per-user RBAC works (end to end)

1. A user logs in at Keycloak (`homelab` realm, `headlamp` confidential client, auth-code flow).
2. Headlamp's backend keeps the user's **`id_token`** and forwards it as the **bearer** on every
   request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour).
3. The apiserver validates that token — it is configured (`infra/ansible` `roles/k3s_server`) with
   `oidc-issuer-url` = this realm, `oidc-client-id=kubernetes`, `oidc-username-claim=email`,
   `oidc-groups-claim=groups`, prefixes `oidc:`.
4. RBAC is applied **per user** via the existing group bindings in
   [`../keycloak/k8s-oidc-rbac.yaml`](../keycloak/k8s-oidc-rbac.yaml):
   `oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`.

So each user gets exactly their own rights — **not** a shared Headlamp identity. That is why the
Headlamp ServiceAccount is **not** bound to cluster-admin: the chart default
`clusterRoleBinding.create: true` (→ `cluster-admin`) is overridden to **`false`** in
`helmrelease.yaml`. User requests ride the forwarded user token; the SA needs no cluster rights.

## The audience nuance (why the client has an audience mapper)

The apiserver only accepts ID tokens whose `aud` **contains `kubernetes`** (`oidc-client-id=
kubernetes`). A plain `headlamp` client issues `aud=[headlamp]`, which the apiserver would reject.
The `headlamp` client therefore carries an **`oidc-audience-mapper`** (see
`../keycloak/keycloak-realm-import.yaml`) with `included.client.audience: kubernetes`, so tokens
carry `aud=[headlamp, kubernetes]`:

- Headlamp's own verifier accepts it (aud contains `headlamp`), and
- the apiserver accepts it (aud contains `kubernetes`).

This keeps the apiserver OIDC config untouched (preferred over widening `--api-audiences`).

## Where the OIDC config comes from (two sources, no overlap)

The container needs five env vars, and their **names are load-bearing**: the chart's args reference
them via `$(...)` (`-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-client-id=$(OIDC_CLIENT_ID)`, …).
They must be exactly `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_ISSUER_URL`, `OIDC_SCOPES`,
`OIDC_CALLBACK_URL` — **not** camelCase. A misnamed or missing name does **not** fail loudly:
Kubernetes leaves the literal string `$(OIDC_ISSUER_URL)` in the argument and login dies with
`unsupported protocol scheme ""`.

| Env var | Source | Why |
|---|---|---|
| `OIDC_CLIENT_ID`, `OIDC_SCOPES`, `OIDC_CLIENT_SECRET` | `headlamp-oidc-credentials.sops.yaml`, `envFrom`'d by `config.oidc.externalSecret` | secret, or environment-invariant |
| `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` | `helmrelease.yaml` → `spec.values.env`, templated `${BASE_DOMAIN}` | **environment-specific and public** |

**Exactly one source per variable.** Never define the same name in both places: an explicit `env:`
entry silently overrides the same name arriving via `envFrom`, so an overlap hides which value is
actually live.

### Why the URLs are not in the Secret (the 2026-09-07 prod outage)

Flux's `${BASE_DOMAIN}` postBuild substitution runs over the **rendered manifest text**; it can never
reach inside an encrypted Secret. `OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` used to be keys in
`headlamp-oidc-credentials.sops.yaml`, so both were frozen at the apex `home.keithjackson.dev`.
Dev **is** the apex, so dev looked fine — but prod (`BASE_DOMAIN = prod.home.keithjackson.dev`) was
pointed at **dev's** Keycloak, `192.168.1.53`, which sits on the dev LAN behind pfSense and is
unroutable from prod's `192.168.2.0/24`. Every prod login failed with:

```
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
dial tcp 192.168.1.53:443: i/o timeout
```

An issuer URL and a callback URL are public, so they now live in cleartext in `helmrelease.yaml`
where substitution works. Only the client secret stays encrypted. The same root cause and the same
fix apply to Semaphore (`catalog/semaphore.yaml`, `SEMAPHORE_OIDC_PROVIDERS`).

**[parity] `OIDC_ISSUER_URL` must equal the kube-apiserver's `oidc-issuer-url`**, which is an
install-time k3s flag set from `k3s_base_domain` (`infra/ansible/roles/k3s_server/defaults/main.yml`,
overridden per inventory group), *not* from `cluster-vars`. Headlamp forwards the user's `id_token`
to the apiserver, so a mismatch 401s every request. Both derive from the same per-environment
domain — dev `home.keithjackson.dev`, prod `prod.home.keithjackson.dev` — which is what keeps them
in step; changing one without the other breaks login.

## Secrets

- The confidential client secret lives SOPS-encrypted in
  [`../keycloak/keycloak-realm-clients.sops.yaml`](../keycloak/keycloak-realm-clients.sops.yaml)
  (`headlamp-client-secret`, injected into the realm import via the `HEADLAMP_CLIENT_SECRET`
  placeholder) and mirrored in this namespace's `headlamp-oidc-credentials.sops.yaml`
  (`OIDC_CLIENT_SECRET`), which the chart `config.oidc.externalSecret` envFrom's. Same cross-namespace
  mirror pattern as grafana/gitea.

## Break-glass

The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full
access if Keycloak/OIDC is down — the standard break-glass path.

## Files

| File | Tier | Purpose |
|---|---|---|
| `namespace.yaml` | platform | `headlamp` namespace (`gateway-access=shared`; platform, not app-namespace) |
| `helmrepository.yaml` / `helmrelease.yaml` | platform | Official `kubernetes-sigs/headlamp` chart + OIDC/RBAC values |
| `headlamp-oidc-credentials.sops.yaml` | platform | SOPS OIDC **secret** the chart envFrom's — client id/scopes/secret only; the per-env URLs live in `helmrelease.yaml` `spec.values.env` |
| `networkpolicy.yaml` | platform | default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows |
| `httproute.yaml` | platform-config | Gateway API HTTPRoute for `headlamp.${BASE_DOMAIN}` (needs Gateway CRDs) |