App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)
Headlamp is served at https://headlamp.home.keithjackson.dev behind the shared Gateway
(wildcard TLS), with native Keycloak OIDC login and per-user RBAC (ADR-0014, docs/15).
How per-user RBAC works (end to end)
- A user logs in at Keycloak (
homelabrealm,headlampconfidential client, auth-code flow). - Headlamp's backend keeps the user's
id_tokenand forwards it as the bearer on every request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour). - The apiserver validates that token — it is configured (
infra/ansibleroles/k3s_server) withoidc-issuer-url= this realm,oidc-client-id=kubernetes,oidc-username-claim=email,oidc-groups-claim=groups, prefixesoidc:. - RBAC is applied per user via the existing group bindings in
../keycloak/k8s-oidc-rbac.yaml:oidc:platform-admins→cluster-admin,oidc:viewers→view.
So each user gets exactly their own rights — not a shared Headlamp identity. That is why the
Headlamp ServiceAccount is not bound to cluster-admin: the chart default
clusterRoleBinding.create: true (→ cluster-admin) is overridden to false in
helmrelease.yaml. User requests ride the forwarded user token; the SA needs no cluster rights.
The audience nuance (why the client has an audience mapper)
The apiserver only accepts ID tokens whose aud contains kubernetes (oidc-client-id= kubernetes). A plain headlamp client issues aud=[headlamp], which the apiserver would reject.
The headlamp client therefore carries an oidc-audience-mapper (see
../keycloak/keycloak-realm-import.yaml) with included.client.audience: kubernetes, so tokens
carry aud=[headlamp, kubernetes]:
- Headlamp's own verifier accepts it (aud contains
headlamp), and - the apiserver accepts it (aud contains
kubernetes).
This keeps the apiserver OIDC config untouched (preferred over widening --api-audiences).
Secrets
- The confidential client secret lives SOPS-encrypted in
../keycloak/keycloak-realm-clients.sops.yaml(headlamp-client-secret, injected into the realm import via theHEADLAMP_CLIENT_SECRETplaceholder) and mirrored in this namespace'sheadlamp-oidc-credentials.sops.yaml(OIDC_CLIENT_SECRET), which the chartconfig.oidc.externalSecretenvFrom's. Same cross-namespace mirror pattern as grafana/gitea. - Key names are load-bearing. The chart
envFroms this Secret verbatim, so its keys become the container env vars referenced by the chart's OIDC args (-oidc-idp-issuer-url=$(OIDC_ISSUER_URL), etc.). They must be exactlyOIDC_CLIENT_ID,OIDC_CLIENT_SECRET,OIDC_ISSUER_URL,OIDC_SCOPES(andOIDC_CALLBACK_URL) — not camelCase. A misnamed key leaves$(OIDC_ISSUER_URL)unexpanded and OIDC login fails withunsupported protocol scheme "".
Break-glass
The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full access if Keycloak/OIDC is down — the standard break-glass path.
Files
| File | Tier | Purpose |
|---|---|---|
namespace.yaml |
platform | headlamp namespace (gateway-access=shared; platform, not app-namespace) |
helmrepository.yaml / helmrelease.yaml |
platform | Official kubernetes-sigs/headlamp chart + OIDC/RBAC values |
headlamp-oidc-credentials.sops.yaml |
platform | SOPS OIDC config the chart envFrom's (decrypted by the platform Flux Kustomization) |
networkpolicy.yaml |
platform | default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows |
httproute.yaml |
platform-config | Gateway API HTTPRoute for headlamp.home.keithjackson.dev (needs Gateway CRDs) |