Files
jakman85andClaude Opus 5 bfa39bc5d0 fix(headlamp): per-environment OIDC URLs — prod was dialing DEV's Keycloak
The operator could not log in to Headlamp on prod:

  Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
  dial tcp 192.168.1.53:443: i/o timeout

ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted
headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites
the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen
at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked
correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak
at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24.

FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the
HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per
environment. The Secret keeps only what is genuinely secret or environment-invariant
(OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES).

Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit
env entry silently WINS over the same name from envFrom — so the two sets must not overlap,
and they no longer do. The five load-bearing names the chart's $(...) args require
(OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered;
verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and
`helm template` of chart 0.41.0 against the substituted values.

Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT
`-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via
`env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL
had been dead weight. The value is byte-identical to the single redirectUri registered on the
Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts.

[parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from
k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 02:06:38 -05:00

132 lines
7.9 KiB
YAML

---
# Headlamp — the Kubernetes web dashboard, served at headlamp.home.keithjackson.dev with Keycloak
# native-OIDC login and PER-USER RBAC (ADR-0014, docs/15). Official kubernetes-sigs/headlamp Helm
# chart via a Flux HelmRelease (same pattern as platform/monitoring's kube-prometheus-stack).
#
# ── PER-USER RBAC (the whole point) ────────────────────────────────────────────────────────────
# Headlamp is configured for OIDC, so its backend forwards the LOGGED-IN USER's OIDC id_token as the
# bearer to the k3s API server. The apiserver validates that token (oidc-issuer-url = this realm,
# oidc-client-id=kubernetes, oidc-username-claim=email, oidc-groups-claim=groups; see
# infra/ansible roles/k3s_server) and RBAC is applied per user via the EXISTING group bindings
# (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins -> cluster-admin, oidc:viewers -> view).
# So every user gets exactly their own rights — NOT a shared identity.
#
# clusterRoleBinding.create=false ← DELIBERATE. The default chart binds Headlamp's ServiceAccount
# to cluster-admin; that would let the backend act as cluster-admin. We do NOT want that: user
# requests ride the user's forwarded id_token, so the SA needs no cluster rights. Removing the
# binding means an unauthenticated backend can do nothing — the secure posture for per-user RBAC.
#
# ── AUDIENCE NUANCE ─────────────────────────────────────────────────────────────────────────────
# The apiserver only accepts ID tokens whose `aud` CONTAINS `kubernetes` (oidc-client-id=kubernetes).
# A `headlamp` client would issue aud=[headlamp]. The Keycloak `headlamp` client therefore carries an
# AUDIENCE protocol mapper that ADDS `kubernetes` to the token audience (keycloak-realm-import.yaml),
# giving aud=[headlamp, kubernetes]: Headlamp's own verifier accepts it (aud contains headlamp) AND
# the apiserver accepts it (aud contains kubernetes). This keeps the apiserver config untouched.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: headlamp
namespace: headlamp
spec:
interval: 30m
chart:
spec:
chart: headlamp
# 0.41.x, NOT 0.40.x. Chart 0.40.1 shipped the `-session-ttl=86400` container arg BEFORE the
# backend flag existed: its image (appVersion v0.40.1) has no `session-ttl` flag, so the pod
# crashlooped with `flag provided but not defined: -session-ttl` (upstream issue #4883). The
# backend gained `f.Int("session-ttl", …)` in v0.41.0, and chart 0.41.0 pins that same image
# (appVersion == chart version), so chart+image realign: every arg the chart renders for our
# values (-session-ttl + the -oidc-* args from the externalSecret; useCookie stays default-false
# so `-oidc-use-cookie` is NOT emitted) is supported by the v0.41.0 backend. Renovate maintains
# within 0.41.x; the appVersion continues to pin the Headlamp image.
version: ">=0.41.0 <0.42.0"
sourceRef:
kind: HelmRepository
name: headlamp
namespace: headlamp
install:
remediation:
retries: 3
upgrade:
remediation:
retries: 3
values:
replicaCount: 1
# In-cluster mode: Headlamp discovers the apiserver from the pod's in-cluster config, then
# OVERRIDES the bearer with the user's OIDC id_token for authenticated requests.
config:
inCluster: true
oidc:
# OIDC config reaches the container as env vars whose NAMES ARE LOAD-BEARING: the chart's
# args reference them via $(...) — `-oidc-client-id=$(OIDC_CLIENT_ID)`,
# `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, `-oidc-callback-url=$(OIDC_CALLBACK_URL)`, … — so
# they MUST be exactly OIDC_CLIENT_ID / OIDC_CLIENT_SECRET / OIDC_ISSUER_URL / OIDC_SCOPES /
# OIDC_CALLBACK_URL. A misnamed or absent name does NOT fail: Kubernetes leaves the LITERAL
# string `$(OIDC_ISSUER_URL)` in the arg and login dies with `unsupported protocol scheme ""`.
#
# There are exactly TWO sources, with NO variable served by both (a name defined in `env:`
# would silently WIN over the same name coming from `envFrom`, so overlap is forbidden here):
# • envFrom this SOPS Secret → OIDC_CLIENT_ID, OIDC_SCOPES, OIDC_CLIENT_SECRET
# • the `env:` block below → OIDC_ISSUER_URL, OIDC_CALLBACK_URL
# The Secret is consumed by name only (R-IDP-04); nothing sensitive is rendered here.
externalSecret:
enabled: true
name: headlamp-oidc
hasScopes: true
# ── PER-ENVIRONMENT OIDC URLs (2026-09-07 fix) ───────────────────────────────────────────────
# These two used to be keys INSIDE headlamp-oidc-credentials.sops.yaml. That was the bug: Flux's
# `${BASE_DOMAIN}` postBuild substitution operates on the RENDERED manifest text and can never
# reach inside an encrypted Secret, so both were frozen at the apex `home.keithjackson.dev`.
# Dev is the apex, so dev looked fine; PROD (BASE_DOMAIN = prod.home.keithjackson.dev) therefore
# pointed Headlamp at DEV's Keycloak — 192.168.1.53, on the dev LAN behind pfSense, unroutable
# from prod's 192.168.2.0/24 — and every login died with
# dial tcp 192.168.1.53:443: i/o timeout on .../.well-known/openid-configuration
# An issuer URL and a callback URL are PUBLIC. They belong in cleartext, where substitution works.
#
# [parity] OIDC_ISSUER_URL MUST equal the kube-apiserver's `oidc-issuer-url`, which is an
# INSTALL-TIME k3s flag derived from `k3s_base_domain` (infra/ansible roles/k3s_server/defaults +
# inventory/group_vars/k3s_<env>/vars.yml), NOT from this ConfigMap. Headlamp forwards the user's
# id_token to the apiserver as the bearer, so an issuer mismatch = every request 401s. Both sides
# are keyed off the same per-environment domain (BASE_DOMAIN == k3s_base_domain), which is what
# keeps them in step — dev home.keithjackson.dev, prod prod.home.keithjackson.dev.
#
# NOTE: naming OIDC_CALLBACK_URL here also makes the chart EMIT `-oidc-callback-url=$(…)`, which
# it previously omitted (the arg is gated on the value being set via `env:`/`config.oidc`, and the
# Secret key alone did not trip that gate — so the Secret's callback URL was dead weight). The
# value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client
# (platform/keycloak/keycloak-realm-import.yaml, also `${BASE_DOMAIN}`-templated), which is the
# only URI Keycloak will accept, so making it explicit cannot change behaviour in either env.
env:
- name: OIDC_ISSUER_URL
value: https://keycloak.${BASE_DOMAIN}/realms/homelab
- name: OIDC_CALLBACK_URL
value: https://headlamp.${BASE_DOMAIN}/oidc-callback
# PER-USER RBAC: do NOT grant Headlamp's ServiceAccount cluster-admin. User requests are
# authorized by their forwarded OIDC token against the existing oidc:* group bindings.
serviceAccount:
create: true
clusterRoleBinding:
create: false
# Hardened pod defaults from the chart (runAsNonRoot, uid 100). Kept explicit for clarity.
securityContext:
runAsNonRoot: true
runAsUser: 100
runAsGroup: 101
privileged: false
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
# NOTE: readOnlyRootFilesystem intentionally left unset (chart default). Enabling it needs a
# writable /tmp (+ plugins dir) emptyDir; deferred as a runtime-verified follow-up hardening.
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
memory: 256Mi
service:
type: ClusterIP
port: 80