jakman85andClaude Opus 5
bfa39bc5d0
The operator could not log in to Headlamp on prod: Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": dial tcp 192.168.1.53:443: i/o timeout ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24. FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per environment. The Secret keeps only what is genuinely secret or environment-invariant (OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES). Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit env entry silently WINS over the same name from envFrom — so the two sets must not overlap, and they no longer do. The five load-bearing names the chart's $(...) args require (OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered; verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and `helm template` of chart 0.41.0 against the substituted values. Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT `-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via `env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL had been dead weight. The value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts. [parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses. Co-Authored-By: Claude Opus 5 <[email protected]>
102 lines
6.1 KiB
Markdown
102 lines
6.1 KiB
Markdown
# Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)
|
|
|
|
Headlamp is served at `https://headlamp.${BASE_DOMAIN}` behind the shared Gateway
|
|
(wildcard TLS), with **native Keycloak OIDC login** and **per-user RBAC** (ADR-0014, docs/15).
|
|
|
|
## How per-user RBAC works (end to end)
|
|
|
|
1. A user logs in at Keycloak (`homelab` realm, `headlamp` confidential client, auth-code flow).
|
|
2. Headlamp's backend keeps the user's **`id_token`** and forwards it as the **bearer** on every
|
|
request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour).
|
|
3. The apiserver validates that token — it is configured (`infra/ansible` `roles/k3s_server`) with
|
|
`oidc-issuer-url` = this realm, `oidc-client-id=kubernetes`, `oidc-username-claim=email`,
|
|
`oidc-groups-claim=groups`, prefixes `oidc:`.
|
|
4. RBAC is applied **per user** via the existing group bindings in
|
|
[`../keycloak/k8s-oidc-rbac.yaml`](../keycloak/k8s-oidc-rbac.yaml):
|
|
`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`.
|
|
|
|
So each user gets exactly their own rights — **not** a shared Headlamp identity. That is why the
|
|
Headlamp ServiceAccount is **not** bound to cluster-admin: the chart default
|
|
`clusterRoleBinding.create: true` (→ `cluster-admin`) is overridden to **`false`** in
|
|
`helmrelease.yaml`. User requests ride the forwarded user token; the SA needs no cluster rights.
|
|
|
|
## The audience nuance (why the client has an audience mapper)
|
|
|
|
The apiserver only accepts ID tokens whose `aud` **contains `kubernetes`** (`oidc-client-id=
|
|
kubernetes`). A plain `headlamp` client issues `aud=[headlamp]`, which the apiserver would reject.
|
|
The `headlamp` client therefore carries an **`oidc-audience-mapper`** (see
|
|
`../keycloak/keycloak-realm-import.yaml`) with `included.client.audience: kubernetes`, so tokens
|
|
carry `aud=[headlamp, kubernetes]`:
|
|
|
|
- Headlamp's own verifier accepts it (aud contains `headlamp`), and
|
|
- the apiserver accepts it (aud contains `kubernetes`).
|
|
|
|
This keeps the apiserver OIDC config untouched (preferred over widening `--api-audiences`).
|
|
|
|
## Where the OIDC config comes from (two sources, no overlap)
|
|
|
|
The container needs five env vars, and their **names are load-bearing**: the chart's args reference
|
|
them via `$(...)` (`-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-client-id=$(OIDC_CLIENT_ID)`, …).
|
|
They must be exactly `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_ISSUER_URL`, `OIDC_SCOPES`,
|
|
`OIDC_CALLBACK_URL` — **not** camelCase. A misnamed or missing name does **not** fail loudly:
|
|
Kubernetes leaves the literal string `$(OIDC_ISSUER_URL)` in the argument and login dies with
|
|
`unsupported protocol scheme ""`.
|
|
|
|
| Env var | Source | Why |
|
|
|---|---|---|
|
|
| `OIDC_CLIENT_ID`, `OIDC_SCOPES`, `OIDC_CLIENT_SECRET` | `headlamp-oidc-credentials.sops.yaml`, `envFrom`'d by `config.oidc.externalSecret` | secret, or environment-invariant |
|
|
| `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` | `helmrelease.yaml` → `spec.values.env`, templated `${BASE_DOMAIN}` | **environment-specific and public** |
|
|
|
|
**Exactly one source per variable.** Never define the same name in both places: an explicit `env:`
|
|
entry silently overrides the same name arriving via `envFrom`, so an overlap hides which value is
|
|
actually live.
|
|
|
|
### Why the URLs are not in the Secret (the 2026-09-07 prod outage)
|
|
|
|
Flux's `${BASE_DOMAIN}` postBuild substitution runs over the **rendered manifest text**; it can never
|
|
reach inside an encrypted Secret. `OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` used to be keys in
|
|
`headlamp-oidc-credentials.sops.yaml`, so both were frozen at the apex `home.keithjackson.dev`.
|
|
Dev **is** the apex, so dev looked fine — but prod (`BASE_DOMAIN = prod.home.keithjackson.dev`) was
|
|
pointed at **dev's** Keycloak, `192.168.1.53`, which sits on the dev LAN behind pfSense and is
|
|
unroutable from prod's `192.168.2.0/24`. Every prod login failed with:
|
|
|
|
```
|
|
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
|
|
dial tcp 192.168.1.53:443: i/o timeout
|
|
```
|
|
|
|
An issuer URL and a callback URL are public, so they now live in cleartext in `helmrelease.yaml`
|
|
where substitution works. Only the client secret stays encrypted. The same root cause and the same
|
|
fix apply to Semaphore (`catalog/semaphore.yaml`, `SEMAPHORE_OIDC_PROVIDERS`).
|
|
|
|
**[parity] `OIDC_ISSUER_URL` must equal the kube-apiserver's `oidc-issuer-url`**, which is an
|
|
install-time k3s flag set from `k3s_base_domain` (`infra/ansible/roles/k3s_server/defaults/main.yml`,
|
|
overridden per inventory group), *not* from `cluster-vars`. Headlamp forwards the user's `id_token`
|
|
to the apiserver, so a mismatch 401s every request. Both derive from the same per-environment
|
|
domain — dev `home.keithjackson.dev`, prod `prod.home.keithjackson.dev` — which is what keeps them
|
|
in step; changing one without the other breaks login.
|
|
|
|
## Secrets
|
|
|
|
- The confidential client secret lives SOPS-encrypted in
|
|
[`../keycloak/keycloak-realm-clients.sops.yaml`](../keycloak/keycloak-realm-clients.sops.yaml)
|
|
(`headlamp-client-secret`, injected into the realm import via the `HEADLAMP_CLIENT_SECRET`
|
|
placeholder) and mirrored in this namespace's `headlamp-oidc-credentials.sops.yaml`
|
|
(`OIDC_CLIENT_SECRET`), which the chart `config.oidc.externalSecret` envFrom's. Same cross-namespace
|
|
mirror pattern as grafana/gitea.
|
|
|
|
## Break-glass
|
|
|
|
The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full
|
|
access if Keycloak/OIDC is down — the standard break-glass path.
|
|
|
|
## Files
|
|
|
|
| File | Tier | Purpose |
|
|
|---|---|---|
|
|
| `namespace.yaml` | platform | `headlamp` namespace (`gateway-access=shared`; platform, not app-namespace) |
|
|
| `helmrepository.yaml` / `helmrelease.yaml` | platform | Official `kubernetes-sigs/headlamp` chart + OIDC/RBAC values |
|
|
| `headlamp-oidc-credentials.sops.yaml` | platform | SOPS OIDC **secret** the chart envFrom's — client id/scopes/secret only; the per-env URLs live in `helmrelease.yaml` `spec.values.env` |
|
|
| `networkpolicy.yaml` | platform | default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows |
|
|
| `httproute.yaml` | platform-config | Gateway API HTTPRoute for `headlamp.${BASE_DOMAIN}` (needs Gateway CRDs) |
|