jakman85andClaude Opus 5
f5931944fa
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa flow in the realm import. That declaration could not take effect: a KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so every realm-level field and flow declared after the first import is inert -- in git, applied by Flux, reported Ready, reaching the realm never. Merging it changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the 2026-09-06 client-secret incident, third field over. A SIBLING reconciler converges all three declared things every 15 minutes. It is a sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's allow-list -- and its guarantee that prod's real user and the media-users / platform-admins / viewers groups are structurally unreachable -- stays literally true. Its selftest still passes unchanged. pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of GET/PUT /admin/realms/homelab, readback-verified. pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE literally-named alias; enabled, defaultAction false so enrolment stays opt-in. pass 3 the browser flow -- BLUE/GREEN. PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a five-flow, ten-execution tree needs create-subflow, create-execution, set-requirement and bind-config as SEPARATE untransacted calls; doing that to the BOUND flow means a run dying between them leaves a half-built login path for every SSO app and kubectl, and this repo has had a total SSO outage from a realm change. So instead: 1. BUILD under a NEW, unused alias (declared alias + a hash of the desired shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow and zero user-visible effect. 2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the conditional-OTP config CONTENTS -- before anything is bound, so the regression that once 403'd every platform-admin is caught BEFORE it bites. 3. BIND with ONE atomic PUT setting browserFlow. 4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old flow survives as the rollback target. The widened allow-list is made safe structurally. It cannot say "but not the bound flow" -- its write entries are alias-parameterised -- so a second guard screens every write against the live browserFlow's WHOLE subtree (sub-flow aliases and execution ids) before a socket opens, FAIL-CLOSED until that binding has been read. The create-flow endpoint carries no alias in its path, so it is screened on its request BODY; the selftest caught that it otherwise escaped the guard. Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the login path on a schedule. The bound flow is compared STRUCTURALLY and a match produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately not compared -- a built flow carries a hash suffix, so comparing labels would make it permanently unrecognisable as converged. Single source of truth: the desired state is GENERATED from the realm import by ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked into every credential at enrolment and never re-keyable -- is declared once. The pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no rbac.yaml is added anywhere. Read from Keycloak 26.0.6 source before being relied on: updateRealm (DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/ getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read getBrowserFlow(), which is what makes the atomic bind possible and why the policy write pins it; the passwordless policy is applied all-or-nothing only when RpEntityName is non-empty, with defaults substituted for omitted keys; and updateExecutions compares rep.getPriority(), so a body omitting `priority` would silently reshuffle the whole execution order. Gate coverage: a new whole-repo L0 check extracts the script and runs its --selftest (216 assertions, including a build killed at four different points each asserting the old flow is still bound and the bound subtree was never written), plus mutations.py (30 injected regressions, all caught) proving the selftest can fail. An earlier iteration of this work only AUDITED the flow and left the restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records that design and why it was reconsidered. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/
Cluster-wide platform services, Flux-managed (R-PLAT-01, R-GITOPS-01/02/03). Reconciled
by the platform Flux Kustomization defined in clusters/base/platform.yaml,
before apps/.
Governing spec: docs/05-gitops-and-delivery.md,
docs/01-architecture-overview.md.
Phase 0 scaffolding only. Every component here is a structurally-correct stub — a Flux HelmRelease and/or Kustomization with a values placeholder, a README citing the governing ADR/spec, and TODOs for the phase that fills it in. No cluster exists; nothing reconciles yet. Do not add working config, secrets, or provisioning under Phase 0 (spec
AGENTS.md§1/§5/§6,roadmap/phases.md).
Reconciliation flow (how a change lands)
git (source of truth)
│ operator PR → CI (lint · schema · Kyverno L0 · Chainsaw) → merge to tracked branch
▼
Flux GitRepository (clusters/base/flux-system.yaml)
▼
Flux Kustomization "platform" → ./platform (this dir) ── substituteFrom cluster-vars ──┐
│ prune=true (drift correction, R-GITOPS-03) · wait=true │ R-ENV-02
▼ │
Flux Kustomization "apps" → ./apps (dependsOn platform) ─ substituteFrom cluster-vars ──┘
Git is authoritative; the cluster conforms to git and drift is auto-corrected (R-GITOPS-01/03).
Manual kubectl apply of platform state is prohibited (spec AGENTS.md §5).
The dev ↔ prod overlay model (R-ENV-02)
Platform manifests are authored once. Every value that legitimately differs between environments
is a ${TOKEN} resolved from the cluster-vars ConfigMap via Flux postBuild.substituteFrom. The
clusters/dev and clusters/prod overlays differ only by the values they patch into
cluster-vars (issuer, replica counts, storage sizes, retention). See
clusters/README.md. Base manifests are never forked per environment.
Components (one directory each)
| Dir | Component | Governing ADR / spec | Serves |
|---|---|---|---|
cilium/ |
CNI + kube-proxy replacement + Gateway API + Egress Gateway + Hubble | ADR-0002, docs/03 | R-NET-* |
longhorn/ |
Replicated block storage; 3-replica default StorageClass + bulk tier | ADR-0003, docs/04 | R-STATE-* |
cert-manager/ |
TLS via Let's Encrypt DNS-01 (Cloudflare); staging + prod ClusterIssuers | ADR-0006, docs/08 | R-TLS-* |
externaldns/ |
Automatic DNS records for public services (Cloudflare) | docs/09 | R-DNS-* |
gateway/ |
Gateway API GatewayClass + shared HTTPS Gateway + wildcard cert | ADR-0002/ADR-0006, docs/09 | R-DNS-01, R-TLS-01 |
monitoring/ |
kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager | docs/10 | R-OBS-* |
velero/ |
Off-cluster backup/DR of volumes + resource state | ADR-0003, docs/04 | R-STATE-03 |
kubevirt/ |
KubeVirt + CDI (IaaS: application VMs on the same K8s surface) | ADR-0005, docs/06 | R-PLAT-03 |
image-automation/ |
Flux image-update-automation (hands-off image tag advance) | ADR-0009, docs/11 | R-UPD-02 |
keycloak/ |
Platform IdP (OIDC + SAML), config-as-code, PostgreSQL on Longhorn | ADR-0014, docs/15 | R-IDP-* |
oauth2-proxy/ |
Forward-auth at the Gateway for non-OIDC apps → Keycloak | ADR-0014, docs/15 | R-IDP-* |
kyverno/ |
L0 policy-as-code gate — OWNED BY THE POLICY ENGINEER (WS-TEST). | ADR-0011, docs/14 | R-TEST-* |
⚠️ INTEGRATION TODO(PM): wire kyverno into the platform kustomization
platform/kyverno/ is authored by a different agent (the policy engineer, WS-TEST). This
workstream did not create any files under platform/kyverno/ and must not.
platform/kustomization.yaml intentionally omits - kyverno. Once the
policy engineer's files land, the PM must add - kyverno to that list (recommended first, so
policies are present before other components reconcile). The line is present as a commented TODO in
the kustomization.
Secrets
Every secret referenced below (Cloudflare API token, Velero backup-target creds, Keycloak admin/DB, oauth2-proxy cookie/client secrets) is referenced by name only and will be committed SOPS-encrypted (WS-SEC, ADR-0007). No plaintext secret appears anywhere in this tree.
Cross-workstream contracts surfaced by these stubs
- cilium (WS-NET ↔ WS-INFRA): Cilium's initial CNI install is done by Ansible at cluster bootstrap (docs/02, Phase 1). The HelmRelease here manages Cilium config thereafter; its Helm values MUST match the Ansible bootstrap values (kubeProxyReplacement, k8sServiceHost/Port, API server host) or reconciliation will fight the bootstrap. Contract owner: WS-NET + WS-INFRA.
- gateway + cert-manager + externaldns (WS-NET + WS-SEC): the shared Gateway listener, the
wildcard
*.home.keithjackson.devcert, and the ExternalDNS source all reference each other; they form the "Issuer + Gateway TLS" interface inroadmap/workstreams.md. - image-automation (WS-UPDATE): this dir holds only the cluster-wide ImageUpdateAutomation +
git write-back. Per-service
ImageRepository/ImagePolicyare rendered from each ServiceDefinition'supdateblock (WS-GITOPS/WS-UPDATE), living withapps/. - keycloak/oauth2-proxy (WS-SEC): the SSO/auth contract (
authblock → Keycloak client / oauth2-proxy protection) is owned by WS-SEC; these are placeholders pending that contract.