jakman85andClaude Opus 4.8
071123ae43
Headlamp OIDC login failed with `unsupported protocol scheme ""`. The kubernetes-sigs/headlamp chart `envFrom`s the headlamp-oidc Secret verbatim and renders args like `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, so the Secret keys must be the exact env var names. The keys were camelCase (clientID/issuerURL/scopes/callbackURL/clientSecret), following the mistaken `.example`, so `OIDC_ISSUER_URL` was never defined. Kubernetes left the literal `$(OIDC_ISSUER_URL)` in the arg and Headlamp's OIDC discovery hit `Get "$(OIDC_ISSUER_URL)/.well-known/openid-configuration": unsupported protocol scheme ""`. Rename the SOPS Secret keys to OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES / OIDC_CALLBACK_URL (values unchanged, still SOPS/age-encrypted). Fix the `.example` and README so the correct, load-bearing key names are documented and the camelCase footgun can't recur. Co-Authored-By: Claude Opus 4.8 <[email protected]>
64 lines
3.8 KiB
Markdown
64 lines
3.8 KiB
Markdown
# Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)
|
|
|
|
Headlamp is served at `https://headlamp.home.keithjackson.dev` behind the shared Gateway
|
|
(wildcard TLS), with **native Keycloak OIDC login** and **per-user RBAC** (ADR-0014, docs/15).
|
|
|
|
## How per-user RBAC works (end to end)
|
|
|
|
1. A user logs in at Keycloak (`homelab` realm, `headlamp` confidential client, auth-code flow).
|
|
2. Headlamp's backend keeps the user's **`id_token`** and forwards it as the **bearer** on every
|
|
request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour).
|
|
3. The apiserver validates that token — it is configured (`infra/ansible` `roles/k3s_server`) with
|
|
`oidc-issuer-url` = this realm, `oidc-client-id=kubernetes`, `oidc-username-claim=email`,
|
|
`oidc-groups-claim=groups`, prefixes `oidc:`.
|
|
4. RBAC is applied **per user** via the existing group bindings in
|
|
[`../keycloak/k8s-oidc-rbac.yaml`](../keycloak/k8s-oidc-rbac.yaml):
|
|
`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`.
|
|
|
|
So each user gets exactly their own rights — **not** a shared Headlamp identity. That is why the
|
|
Headlamp ServiceAccount is **not** bound to cluster-admin: the chart default
|
|
`clusterRoleBinding.create: true` (→ `cluster-admin`) is overridden to **`false`** in
|
|
`helmrelease.yaml`. User requests ride the forwarded user token; the SA needs no cluster rights.
|
|
|
|
## The audience nuance (why the client has an audience mapper)
|
|
|
|
The apiserver only accepts ID tokens whose `aud` **contains `kubernetes`** (`oidc-client-id=
|
|
kubernetes`). A plain `headlamp` client issues `aud=[headlamp]`, which the apiserver would reject.
|
|
The `headlamp` client therefore carries an **`oidc-audience-mapper`** (see
|
|
`../keycloak/keycloak-realm-import.yaml`) with `included.client.audience: kubernetes`, so tokens
|
|
carry `aud=[headlamp, kubernetes]`:
|
|
|
|
- Headlamp's own verifier accepts it (aud contains `headlamp`), and
|
|
- the apiserver accepts it (aud contains `kubernetes`).
|
|
|
|
This keeps the apiserver OIDC config untouched (preferred over widening `--api-audiences`).
|
|
|
|
## Secrets
|
|
|
|
- The confidential client secret lives SOPS-encrypted in
|
|
[`../keycloak/keycloak-realm-clients.sops.yaml`](../keycloak/keycloak-realm-clients.sops.yaml)
|
|
(`headlamp-client-secret`, injected into the realm import via the `HEADLAMP_CLIENT_SECRET`
|
|
placeholder) and mirrored in this namespace's `headlamp-oidc-credentials.sops.yaml`
|
|
(`OIDC_CLIENT_SECRET`), which the chart `config.oidc.externalSecret` envFrom's. Same cross-namespace
|
|
mirror pattern as grafana/gitea.
|
|
- **Key names are load-bearing.** The chart `envFrom`s this Secret verbatim, so its keys become the
|
|
container env vars referenced by the chart's OIDC args (`-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`,
|
|
etc.). They must be exactly `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_ISSUER_URL`, `OIDC_SCOPES`
|
|
(and `OIDC_CALLBACK_URL`) — **not** camelCase. A misnamed key leaves `$(OIDC_ISSUER_URL)`
|
|
unexpanded and OIDC login fails with `unsupported protocol scheme ""`.
|
|
|
|
## Break-glass
|
|
|
|
The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full
|
|
access if Keycloak/OIDC is down — the standard break-glass path.
|
|
|
|
## Files
|
|
|
|
| File | Tier | Purpose |
|
|
|---|---|---|
|
|
| `namespace.yaml` | platform | `headlamp` namespace (`gateway-access=shared`; platform, not app-namespace) |
|
|
| `helmrepository.yaml` / `helmrelease.yaml` | platform | Official `kubernetes-sigs/headlamp` chart + OIDC/RBAC values |
|
|
| `headlamp-oidc-credentials.sops.yaml` | platform | SOPS OIDC config the chart envFrom's (decrypted by the `platform` Flux Kustomization) |
|
|
| `networkpolicy.yaml` | platform | default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows |
|
|
| `httproute.yaml` | platform-config | Gateway API HTTPRoute for `headlamp.home.keithjackson.dev` (needs Gateway CRDs) |
|