Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
Files
f5931944fabb0a06dd376276b49ea4f687a50c8f
home-network-k3s-impl/platform-config/kustomization.yaml
T
jakman85andClaude Opus 5 f5931944fa feat(keycloak): reconcile the passkey realm config via GitOps, the browser flow blue/green 
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.

A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.

  pass 1  webAuthnPolicyPasswordless* -- strict read-modify-write of
          GET/PUT /admin/realms/homelab, readback-verified.
  pass 2  webauthn-register-passwordless -- strict read-modify-write of ONE
          literally-named alias; enabled, defaultAction false so enrolment stays
          opt-in.
  pass 3  the browser flow -- BLUE/GREEN.

PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:

  1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
     shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
     mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
     and zero user-visible effect.
  2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
     conditional-OTP config CONTENTS -- before anything is bound, so the
     regression that once 403'd every platform-admin is caught BEFORE it bites.
  3. BIND with ONE atomic PUT setting browserFlow.
  4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
     flow survives as the rollback target.

The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.

Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.

Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.

Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.

Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-29 21:22:08 -05:00

165 lines
12 KiB
YAML
Raw Blame History

---
# platform-config — the CONFIGS half of the controllers/configs split (flux2 idiomatic).
#
# WHY: the `platform` Flux Kustomization installs the CONTROLLERS (HelmReleases) that register CRDs
# (cert-manager.io, kyverno.io, …). The custom resources BELOW use those CRDs, so applying them in
# the SAME pass fails atomically ("no matches for kind ClusterIssuer") and blocks the whole platform
# apply. This Kustomization carries the CRs and is gated `dependsOn: platform` (wait) so the CRDs
# exist first. See clusters/base/platform-config.yaml + DECISIONS-LOG.
#
# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
# Kyverno policies must stay under platform/kyverno/ for the CI L0 `kyverno test` gate). Build/verify
# locally with: kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# cert-manager: DNS-01 issuers (need cert-manager.io CRDs from the cert-manager HelmRelease)
- ../platform/cert-manager/clusterissuers.yaml
# gateway: the Gateway + wildcard Certificate (Certificate needs cert-manager.io; Gateway needs the
# GatewayClass, which stays in platform/gateway as a controller resource)
- ../platform/gateway/gateway.yaml
- ../platform/gateway/wildcard-certificate.yaml
# kyverno: ClusterPolicies (need kyverno.io CRDs from the Kyverno HelmRelease)
- ../platform/kyverno/policies/require-default-deny-baseline.yaml
- ../platform/kyverno/policies/validate-default-deny-shape.yaml
- ../platform/kyverno/policies/restrict-host-namespaces.yaml
- ../platform/kyverno/policies/require-egress-class-label.yaml
- ../platform/kyverno/policies/require-auth-mode-label.yaml
# COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label
# policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*,
# cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them
# and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds
# the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED
# rather than bypassed. Applying it here (not later) matters: if it lands after the validating
# policies, the next import is blocked in the gap.
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
- ../platform/kyverno/admission/secret-provenance.yaml
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
# only ("kept as reference; THIS file is the buildable render root" in
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
# leaving the conformance-trigger identity live with nothing bounding it.
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
# R-SEC-02 — bounds what the PM agent dispatcher may create (platform/pm-agent/). SAME
# REASON THIS LINE EXISTS AS THE ONE ABOVE, and the same consequence if it is omitted: this
# Kustomization is the only list that applies ClusterPolicies, so adding a policy to
# platform/kyverno/admission/kustomization.yaml alone is a SILENT NO-OP. The policy would
# pass `kyverno test`, pass the gate, and never reach the cluster — leaving a
# ServiceAccount that can create Jobs in a namespace, driven by a model reading untrusted
# issue text, with nothing bounding the shape of what it creates.
- ../platform/kyverno/admission/pm-agent-job-shape.yaml
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
- cilium/l2-ip-pool.yaml
- cilium/l2-announcement-policy.yaml
# longhorn/external-snapshotter: the driver.longhorn.io VolumeSnapshotClass (needs the
# snapshot.storage.k8s.io CRDs from platform/external-snapshotter). Enables CSI snapshots +
# CDI snapshot smart-clone.
- ../platform/longhorn/volumesnapshotclass.yaml
# monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
- ../platform/monitoring/alerts.yaml
# Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule
# (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC
# was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
# file header. (2026-09-06, from the CNPG WAL/archiving incident.)
- ../platform/monitoring/pvc-capacity-alerts.yaml
# PodMonitor for the six Flux controllers — the ONLY thing that puts `gotk_*` in the TSDB.
# kube-prometheus-stack does not scrape flux-system (the controllers are bootstrapped, expose no
# Service, and no chart value covers them), so until this landed the five gotk_* rules in
# alerts.yaml above were structurally inert: verified on prod 2026-09-13,
# `count({__name__=~"gotk.*"})` returned nothing. A CR of monitoring.coreos.com -> this tier, same
# as alerts.yaml. (2026-09-13, from the inert-alert audit.)
- ../platform/monitoring/flux-podmonitor.yaml
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
# operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
# installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
# ConfigMap kept beside them for locality.
- ../platform/kubevirt/kubevirt-cr.yaml
- ../platform/kubevirt/cdi-cr.yaml
# *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
# The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
# (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
#
# `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
# `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
# only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
# in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
# `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
# created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
# A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
# "cdi.kubevirt.io/v1beta1"
# Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
#
# THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
# is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
- ../platform/kubevirt/servicemonitor.yaml
- ../platform/kubevirt/alerts.yaml
- ../platform/kubevirt/dashboard.yaml
# --- Phase 6 — Identity / SSO (ADR-0014) ---
# keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
# operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
# ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
# cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a
# CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass —
# identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG
# clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted.
- ../platform/cnpg/alerts.yaml
- ../platform/keycloak/postgres-cluster.yaml
- ../platform/keycloak/keycloak-instance.yaml
- ../platform/keycloak/keycloak-realm-import.yaml
# The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is
# stored LITERALLY and the import still reports success, then rests at Done=True and
# never re-runs. This CronJob is the level-triggered loop that corrects that drift
# (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform,
# so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and
# alongside the import whose failure mode it covers.
- ../platform/keycloak/client-secret-reconciler.yaml
# The DESIRED STATE for every environment-derived client URL (redirectUris, webOrigins,
# rootUrl, baseUrl, adminUrl), GENERATED from keycloak-realm-import.yaml by
# ci/render-keycloak-client-urls.py and gated by the L0 check. The reconciler pod has no
# Kubernetes API access (automountServiceAccountToken: false), so it cannot read the
# KeycloakRealmImport CR — its desired state must be MOUNTED, hence this ConfigMap.
#
# IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above: both are then
# substituted in one pass by this Kustomization's postBuild.substituteFrom (cluster-vars),
# so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
# cannot disagree about the domain.
- ../platform/keycloak/client-urls.yaml
# PASSKEYS — the same one-shot failure, for the WebAuthn passwordless policy and the
# browser login flow. Measured on prod 2026-10-18, hours after the passkey declaration
# merged: the KeycloakRealmImport read Done=True / Started=False with the whole
# webAuthnPolicyPasswordless* block and the restructured browser-mfa flow INERT. This
# CronJob converges the policy and the enrolment required action, and AUDITS the browser
# flow without the ability to rewrite it (no flow-mutating endpoint is on its
# allow-list) — a half-rewritten flow bound as the realm's browserFlow is a total SSO
# outage, and this repo has had one. See the file header for the full rationale.
- ../platform/keycloak/passkey-reconciler.yaml
# Its DESIRED STATE, GENERATED from keycloak-realm-import.yaml by
# ci/render-keycloak-passkey-policy.py and gated by the L0 check. Same reason as
# client-urls.yaml above: the reconciler pod has no Kubernetes API access
# (automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR
# and its input must be MOUNTED.
#
# IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above, for exactly
# the reason given for client-urls.yaml: webAuthnPolicyPasswordlessRpId is the
# ${BASE_DOMAIN} token, and a split across Kustomizations would let one token resolve to
# two different values. An RP ID is baked into every passkey at enrolment and can never
# be re-keyed, so a domain the two files disagree about is an unrecoverable credential.
- ../platform/keycloak/passkey-desired-state.yaml
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
- ../platform/keycloak/httproute.yaml
- ../platform/keycloak/servicemonitor.yaml
# oauth2-proxy: forward-auth httproute + ServiceMonitor MOVED OUT of platform-config. They now
# travel with the oauth2-proxy controller in its own post-platform Flux Kustomization
# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). Keeping them here — while the
# controller sat in the gating `platform` tier — was one half of the twice-proven cold-start
# deadlock (platform never Ready -> platform-config never reconciles -> these never apply).
# monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR)
- ../platform/monitoring/grafana-httproute.yaml
# headlamp: httproute MOVED OUT of platform-config. It now travels with the headlamp component in
# its own post-platform Flux Kustomization (clusters/base/headlamp.yaml, dependsOn platform-config).
# Keeping it here — while the HelmRelease sat in the gating `platform` tier — meant a headlamp
# crashloop stalled platform-config (and everything below it). See DECISIONS-LOG "Headlamp gating stall".
Reference in new issue View Git Blame Copy Permalink