Files
home-network-k3s-impl/platform/headlamp
jakman85andClaude Opus 4.8 68690e5019 fix(net): allow SSO apps egress to keycloak backend (Cilium 1.20 CVE-2025-30162)
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.

Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-30 23:34:03 -05:00
..

Headlamp — Kubernetes dashboard (Keycloak OIDC + per-user RBAC)

Headlamp is served at https://headlamp.home.keithjackson.dev behind the shared Gateway (wildcard TLS), with native Keycloak OIDC login and per-user RBAC (ADR-0014, docs/15).

How per-user RBAC works (end to end)

  1. A user logs in at Keycloak (homelab realm, headlamp confidential client, auth-code flow).
  2. Headlamp's backend keeps the user's id_token and forwards it as the bearer on every request it proxies to the k3s API server (this is Headlamp's default OIDC behaviour).
  3. The apiserver validates that token — it is configured (infra/ansible roles/k3s_server) with oidc-issuer-url = this realm, oidc-client-id=kubernetes, oidc-username-claim=email, oidc-groups-claim=groups, prefixes oidc:.
  4. RBAC is applied per user via the existing group bindings in ../keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins → cluster-admin, oidc:viewers → view.

So each user gets exactly their own rights — not a shared Headlamp identity. That is why the Headlamp ServiceAccount is not bound to cluster-admin: the chart default clusterRoleBinding.create: true (→ cluster-admin) is overridden to false in helmrelease.yaml. User requests ride the forwarded user token; the SA needs no cluster rights.

The audience nuance (why the client has an audience mapper)

The apiserver only accepts ID tokens whose aud contains kubernetes (oidc-client-id= kubernetes). A plain headlamp client issues aud=[headlamp], which the apiserver would reject. The headlamp client therefore carries an oidc-audience-mapper (see ../keycloak/keycloak-realm-import.yaml) with included.client.audience: kubernetes, so tokens carry aud=[headlamp, kubernetes]:

  • Headlamp's own verifier accepts it (aud contains headlamp), and
  • the apiserver accepts it (aud contains kubernetes).

This keeps the apiserver OIDC config untouched (preferred over widening --api-audiences).

Secrets

  • The confidential client secret lives SOPS-encrypted in ../keycloak/keycloak-realm-clients.sops.yaml (headlamp-client-secret, injected into the realm import via the HEADLAMP_CLIENT_SECRET placeholder) and mirrored in this namespace's headlamp-oidc-credentials.sops.yaml (OIDC_CLIENT_SECRET), which the chart config.oidc.externalSecret envFrom's. Same cross-namespace mirror pattern as grafana/gitea.
  • Key names are load-bearing. The chart envFroms this Secret verbatim, so its keys become the container env vars referenced by the chart's OIDC args (-oidc-idp-issuer-url=$(OIDC_ISSUER_URL), etc.). They must be exactly OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_ISSUER_URL, OIDC_SCOPES (and OIDC_CALLBACK_URL) — not camelCase. A misnamed key leaves $(OIDC_ISSUER_URL) unexpanded and OIDC login fails with unsupported protocol scheme "".

Break-glass

The cert-based admin in the bootstrap kubeconfig is unaffected by any of this and retains full access if Keycloak/OIDC is down — the standard break-glass path.

Files

File Tier Purpose
namespace.yaml platform headlamp namespace (gateway-access=shared; platform, not app-namespace)
helmrepository.yaml / helmrelease.yaml platform Official kubernetes-sigs/headlamp chart + OIDC/RBAC values
headlamp-oidc-credentials.sops.yaml platform SOPS OIDC config the chart envFrom's (decrypted by the platform Flux Kustomization)
networkpolicy.yaml platform default-deny baseline + Cilium ingress(gateway)/egress(apiserver+Keycloak) allows
httproute.yaml platform-config Gateway API HTTPRoute for headlamp.home.keithjackson.dev (needs Gateway CRDs)