Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5.5 6bb8af1d70 docs(decisions): handoff for the passkey reconciler review fixes
gate / gate (pull_request) Failing after 2m55s
gate / conformance (pull_request) Has been skipped
Also states in bind_browser_flow()'s docstring that it is reached only when git's
bind-browser-flow is "true".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:45:40 -05:00
jakman85andClaude Opus 5.5 7c4eb439b2 feat(monitoring): alert when the keycloak passkey reconciler's Job fails
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:41:15 -05:00
jakman85andClaude Opus 5.5 17eed6e8ae fix(keycloak-passkey): bind only on git's say-so; protect real sub-flows; screen the requirement PUT by body id
Review findings on the passkey reconciler, fixed:

1. HIGH - the bind was automatic and rollback did not stick. A git-declared switch,
   `bind-browser-flow` in passkey-desired-state.yaml, now decides whether pass 3 may
   PUT browserFlow. Default "false": build + verify the candidate (or reuse a verified
   one), log WOULD BIND <alias>, never touch the binding - including after an earlier
   bind, so an operator's rollback sticks, and a hand-edited bound candidate is left
   alone rather than shadowed by an -r2 build. "true" binds as before. The renderer
   carries the one hand-edited line through and --check refuses anything but the
   quoted strings "true"/"false". --no-bind can only narrow it.

2. HIGH - the protected-subtree guard read `alias` on sub-flow rows. Keycloak 26.0.6
   AuthenticationManagementResource.recurseExecutions sets displayName =
   flowRef.getAlias() and flowId on a sub-flow row and never sets alias (alias is the
   CONFIG alias on authenticator rows), so the protected set was only the top-level
   alias. Sub-flows are now named by displayName (getFlows() lists top-level flows
   only, so flowId cannot be resolved through it); a sub-flow row without one leaves
   the subtree undetermined and the flow pass fail-closed. The selftest stub now
   returns 26.0.6-shaped rows, and a real run is asserted to refuse writes into
   browser-mfa-forms and browser-mfa-conditional-otp.

3. MEDIUM - PUT /flows/{alias}/executions was screened on the path alias only, but
   updateExecutions resolves the execution realm-wide by body id. body.id is now
   screened against the protected ids AND required to be in the tree this run is
   building (recorded from the build's own readback); the config POST is held to the
   same build scope; and the body may carry only id/requirement/priority (the same
   handler renames any flow named by flowId).

4. Stale "audit-only" docs rewritten (platform-config kustomization, chainsaw test,
   desired-state header and its renderer template, README) to describe what ships.

5. backoffLimit 0 (a retried deterministic build failure burns the candidate-alias
   budget); the stale "Renovate does not manage image: tags" comment corrected and the
   image pinned by index digest.

6. The ~160-line header duplicated in the YAML and the script is now kept once, in the
   script, with a short pointer above it.

Selftest: 259 assertions (was 216). mutations.py: 40 injected regressions (was 30),
including one per finding above, all caught.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:41:14 -05:00
jakman85andClaude Opus 5 f5931944fa feat(keycloak): reconcile the passkey realm config via GitOps, the browser flow blue/green
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.

A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.

  pass 1  webAuthnPolicyPasswordless* -- strict read-modify-write of
          GET/PUT /admin/realms/homelab, readback-verified.
  pass 2  webauthn-register-passwordless -- strict read-modify-write of ONE
          literally-named alias; enabled, defaultAction false so enrolment stays
          opt-in.
  pass 3  the browser flow -- BLUE/GREEN.

PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:

  1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
     shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
     mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
     and zero user-visible effect.
  2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
     conditional-OTP config CONTENTS -- before anything is bound, so the
     regression that once 403'd every platform-admin is caught BEFORE it bites.
  3. BIND with ONE atomic PUT setting browserFlow.
  4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
     flow survives as the rollback target.

The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.

Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.

Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.

Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.

Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-29 21:22:08 -05:00
jakman85 1301af9a4c Merge pull request 'fix(renovate): the config was INVALID, so the first manual run errored out' (#79) from fix/renovate-invalid-config into main
gate / gate (push) Successful in 1m55s
gate / conformance (push) Has been skipped
renovate / renovate (push) Failing after 9s
Reviewed-on: #79
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:29:34 +00:00
jakman85 42ce001d71 Merge pull request 'fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR' (#77) from fix/renovate-cron-window into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #77
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:28:59 +00:00
jakman85 84bb0f0acd Merge pull request 'fix(pihole): make the split-horizon answer cacheable (local-ttl=60)' (#78) from fix/pihole-local-ttl into main
gate / gate (push) Successful in 1m53s
gate / conformance (push) Has been skipped
Reviewed-on: #78
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 08:19:28 +00:00
jakman85andClaude Opus 5 0d4bd7297d fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR
gate / conformance (pull_request) Has been skipped
gate / gate (pull_request) Successful in 1m55s
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.

  03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
  window     ->  03:00-06:00 local   =>  NOT INSIDE (3h45m early)

The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.

Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
  EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
  EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.

Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.

STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:12:28 -05:00
jakman85andClaude Opus 5 713b0c3dba fix(renovate): the config was INVALID, so the first manual run errored out
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
The operator ran the Renovate workflow manually and it failed. The cause is not
the workflow: renovate.json was never a valid Renovate config. It carried five
pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment,
_ignorePaths_comment, _docker_comment) because JSON has no comments, and
Renovate REJECTS unknown top-level options. Proven with the official validator:

  ERROR: Found errors in configuration
    Invalid configuration option: _comment
    Invalid configuration option: _docker_comment
    Invalid configuration option: _ignorePaths_comment
    Invalid configuration option: _kubernetes_comment
    Invalid configuration option: _unmanageable_comment

This is the SECOND independent reason Renovate never worked, on top of PR #77's
cron firing outside renovate.json's own schedule window. Either alone was enough
to produce "nothing ever happens"; #73 shipped both.

Converted renovate.json -> renovate.json5. JSON5 permits real // comments and
Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the
old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original
rationale is preserved as positioned comments (74 comment lines) rather than
flattened into a description string.

Verified the conversion dropped nothing: 18 non-comment keys before and after,
no missing/extra keys, no changed values, docker: {enabled: true} still present.
The validator now returns "Config validated successfully".

DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of
needing to be `custom.regex` (which would silently disable every inline
`# renovate:` marker). The validator accepts `regex`; unfounded, unchanged.

Renovate has still never completed a run -- this fixes one of two blockers and
nothing in the gate catches an invalid config. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:10:25 -05:00
jakman85andClaude Opus 5 fda5c31a94 fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 02:56:49 -05:00
jakman85 c340bbfc4b Merge pull request 'fix(conformance): one regex made the merge gate take 28 minutes instead of 2' (#76) from fix/ha-check-catastrophic-regex into main
gate / gate (push) Successful in 1m56s
gate / conformance (push) Has been skipped
Reviewed-on: #76
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 07:51:51 +00:00
jakman85 a9a529750d Merge pull request 'fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer' (#75) from fix/unattended-upgrades-debian-sentinel into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #75
2026-10-18 07:40:20 +00:00
jakman85andClaude Opus 5 793c3940a8 fix(conformance): one regex made the merge gate take 28 minutes instead of 2
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
ha-control-plane-as-code/check.sh had:

  re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:...'secondary'",
            K3S_TEXT, re.S)

With re.S every `.` matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?`
can each consume the whole document -- nested unbounded repetition, which
backtracks exponentially on a failed or late match. MEASURED in CI via
`docker top` on the live job container: python3 at 19:54 of CPU time, which is
essentially the entire ~28 minute gate.

Localised to the exact line by extracting the inline python and running it under
faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from
this call -- not inferred from reading.

Replaced `.` with `[^\n]*` and dropped re.S, so the {0,10} is a real ten-line
window with nothing to backtrack into. On the real input (22,179 chars):
0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical
success message.

WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every *.yml in
roles/k3s_server concatenated. The regex was latent for as long as it existed;
PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing
the input past the point where exponential backtracking stopped being
survivable.

WHY ONLY infra/** WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so
the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which
selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and
#72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran
~28m.

The pattern is unique in the repo (grep over all l0-static check.sh files).
Not re-timed in CI post-fix; this PR's own run is the first datapoint.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 02:38:23 -05:00
jakman85andClaude Opus 5 ef8ef5ac29 fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer
gate / gate (pull_request) Successful in 27m51s
gate / conformance (pull_request) Has been skipped
The first real run failed on all three nodes:

  fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.

Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.

So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
  running kernel is not the newest installed, using `linux-version` (from
  linux-base, a dependency of every linux-image, measured present) because
  kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
  it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.

The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.

Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.

Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 01:43:43 -05:00
jakman85 ee196c7bcd Merge pull request 'feat(upgrades): automate node reboots (kured) and wire Renovate, which had never run' (#73) from feat/automated-upgrades into main
gate / gate (push) Successful in 27m46s
gate / conformance (push) Has been skipped
Reviewed-on: #73
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 06:28:30 +00:00
jakman85 83d3568150 Merge pull request 'feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet' (#71) from feat/graceful-cluster-shutdown into main
gate / gate (push) Successful in 27m39s
gate / conformance (push) Has been skipped
Reviewed-on: #71
2026-10-18 05:51:20 +00:00
jakman85 24bea918ab Merge pull request 'fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment' (#74) from fix/coredns-ha into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #74
2026-10-18 05:50:46 +00:00
jakman85andClaude Opus 5 5642881c4d fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster
DNS was a single pod on a single node, so draining or rebooting that node was a
CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service
lookup, every CNPG replica finding its primary, every operator addressing the
API by name. It was invisible because 1/1 Ready looks healthy.

It is urgent rather than latent now: the scheduled reboot automation on
feat/automated-upgrades drains and reboots one node at a time, unattended,
overnight -- turning a single point of failure into a recurring one.

WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns
Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml,
managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash
label. k3s re-stages its bundled manifests on startup, so replicas: 2 written
there -- or a kubectl scale -- is reverted on the next restart or upgrade.

Rejected alternatives:
* coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern).
  Deterministic, but takes ownership of coredns from k3s permanently so k3s
  upgrades stop updating it. Too high a price for a replica count.
* A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy
  controller would both own one object and fight -- the default-deny
  NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit.

An HPA is a separate object k3s does not own, driving /scale rather than the
Deployment spec, so there is no shared-ownership fight; if k3s re-stages
replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief
window at 1 replica after a k3s restart.

It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the
autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%),
so the 80% target is never approached. scaleDown stabilization 600s because DNS
is the worst thing to flap; scaleUp immediate, since that restores redundancy.

The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds
the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched.

UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on
restart is reasoned, not observed. After merge check `describe hpa coredns` for
ScalingActive=True -- an inert HPA looks identical to a working one because the
hand scale already left 2 replicas. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 00:44:09 -05:00
jakman85 4d489f2059 Merge pull request 'feat(keycloak): add passkey (WebAuthn passwordless) login, additive and optional' (#72) from feat/keycloak-passkeys into main
gate / gate (push) Successful in 1m53s
gate / conformance (push) Has been skipped
Reviewed-on: #72
2026-10-18 05:44:03 +00:00
jakman85andClaude Opus 5 16894b7b73 feat(keycloak): declare passkey (WebAuthn passwordless) login for the homelab realm
gate / gate (pull_request) Successful in 1m53s
gate / conformance (pull_request) Has been skipped
Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.

Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).

Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.

No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.

UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-17 23:23:36 -05:00
jakman85andClaude Opus 5 2f2e98aa19 feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet
gate / gate (pull_request) Successful in 27m44s
gate / conformance (pull_request) Has been skipped
All three nodes had to be powered off to move them onto a UPS, and there was
NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the
decommissioned dev cluster. The sequence was improvised live. Two defects came
out of doing it.

1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No
   shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s,
   containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG
   instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes.
   roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and
   shutdown-grace-period-critical-pods=20s.

   The half that silently defeats the other half: this is enforced by LOGIND,
   not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits
   at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless,
   truncating the drain with no error. The role ships a logind.conf.d drop-in
   (120) and ASSERTS cap >= grace period, because the failure is invisible.

2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki
   carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the
   quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so
   that PVC was the only copy; 72h retention bounded the loss, the mechanism
   does not. Symptom was the volume count going 25 -> 24, first misread as a
   miscount. Repaired: scale back to the declared 1 recreated the PVC and
   loki-0 is 2/2 Running.

New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence
that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays
running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats
a deletion. Discovery is computed from the live cluster, so a newly onboarded
service needs no edit here.

runbooks/cluster-maintenance-shutdown.md is the operator page: order in both
directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the
un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is
fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding
volume and the self-recovering keycloak-operator from the one thing that needs
a human (a stuck oauth2-proxy replica).

THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s
restart, which is an operator action. Review config.yaml's kubelet-arg before
applying: a bad config.yaml is how k3s fails to come up. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-17 22:26:47 -05:00
36 changed files with 8220 additions and 117 deletions

No files matched your search

+20 -6
View File
@@ -74,12 +74,26 @@ name: renovate
on:
schedule:
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
# of the repo window means a run is always in flight when the window opens.
- cron: "15 3 * * *"
# 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
# `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
# it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
# merely near it.
#
# WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
# run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
# 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
# Every nightly run would have started, logged "not within schedule", opened zero PRs and
# exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
# green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
# needed updating".
#
# THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
# EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
# EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
# 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
# schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
# early-morning-UTC value without redoing this arithmetic.
- cron: "15 8 * * *"
workflow_dispatch:
inputs:
dry_run:
+1 -1
View File
@@ -73,7 +73,7 @@ spec:
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password
valueFrom:
secretKeyRef:
+32 -1
View File
@@ -143,8 +143,39 @@ env:
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
# every client that takes the ADVERTISED connection cannot reach the server at all.
#
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
#
# dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
# dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
# dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
#
# dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
# bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
# * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
# single lookup, which is maximum exposure to any momentary gap; while
# * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
# upstream (standard forwarder behaviour when the forward target does not answer) it latches
# the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
# The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
# live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
# flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
# That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
#
# 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
# re-election (both seconds), short enough that a genuine VIP change still propagates inside a
# minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
# propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
# LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
# the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
# cache expires, the router falls back, and the public answer latches again. That is a real
# outage, and the fix for it is PiHole availability, not DNS caching.
#
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
- name: FTLCONF_misc_dnsmasq_lines
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
# carries the full rationale and the measurement that found it.
+490
View File
@@ -0,0 +1,490 @@
#!/usr/bin/env python3
# =============================================================================
# ci/render-keycloak-passkey-policy.py
#
# Render platform/keycloak/passkey-desired-state.yaml FROM
# platform/keycloak/keycloak-realm-import.yaml.
#
# WHY THIS EXISTS
# ---------------
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
# after the passkey declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
# Done=True Started=False HasErrors=False
#
# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
# is the level-triggered loop that closes that; this renderer supplies the DESIRED
# STATE it converges towards.
#
# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
# flow in a browser; make it the login path" -- not a fact derivable from the realm
# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
# file: --write preserves it, --check accepts any value of it that is exactly the
# string "true" or "false" and rejects everything else (an unquoted YAML true is a
# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
# Kustomization -- so it is refused HERE, at merge time). An absent file or key
# renders as "false". Going live is therefore a one-line PR flipping that line.
#
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
# automountServiceAccountToken: false, exactly like its client-secret sibling. It
# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
# must be MOUNTED. Hence a ConfigMap.
#
# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
# the user-verification requirement and the whole flow tree — free to disagree with
# the realm import. "Two declarations that can disagree" is the same shape as the
# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
# import and the generation is GATED:
# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
# with --check on every merge, and the merge fails if the committed file is not what
# this renderer produces. Same render==golden idiom as
# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
#
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
# rendered by the SAME Flux Kustomization (./platform-config, whose
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
# pass with the same value and cannot disagree about what the base domain is. This
# renderer never parses, splits or reconstructs a hostname and never reads
# cluster-vars — there is no second place the domain is decided.
#
# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
# -----------------------------------------------------
# The enrolment required action, `webauthn-register-passwordless`. It is
# deliberately NOT declared in the realm import, and the import's own header
# explains at length why declaring it there would be actively DANGEROUS: Keycloak
# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
# so declaring one action imports exactly that action plus delete_account and skips
# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
# lock-out of every platform-admin, because auth-otp-form in the browser flow still
# demands a TOTP they could then never enrol.
#
# So the required action cannot be copied from the import, because it is correctly
# absent from it. It is instead DERIVED, as a consequence of what the import DOES
# declare:
#
# IF the declared browser flow contains a `webauthn-authenticator-passwordless`
# execution, THEN passkey logins are offered, THEREFORE the enrolment action must
# be registered and enabled — and NOT a defaultAction, because the import's whole
# stated posture is "additive and optional" and a defaultAction forces enrolment
# on every user at next login.
#
# This is a derivation, not a second declaration: there is no value here that a
# human could edit into disagreement with the import. If the import stops declaring
# the passkey execution, this renderer stops emitting the required action (and
# fails, because then the whole desired state is empty of passkey intent).
#
# WHAT IS DELIBERATELY NOT EXPORTED
# * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
# `webauthn-authenticator` two-factor authenticator; the passwordless
# authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
# into the two-factor family is a silent no-op. Exporting it would hand the
# reconciler a write surface with no purpose.
# * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
# SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
# and users/groups/roles are nobody's — prod's real user and the media-users /
# platform-admins / viewers groups must survive every run of everything.
# * execution `priority`. It is consumed here to ORDER the exported executions
# (Keycloak returns them in priority order), but the numbers themselves are not
# exported: the reconciler compares the resulting SEQUENCE, which is what
# actually determines behaviour. Exporting the integers would make a
# priority renumbering that changes no ordering read as drift.
#
# USAGE
# python3 ci/render-keycloak-passkey-policy.py # print the rendered file
# python3 ci/render-keycloak-passkey-policy.py --write # write it (regenerate)
# python3 ci/render-keycloak-passkey-policy.py --check # exit 1 if it has drifted
# =============================================================================
from __future__ import annotations
import argparse
import difflib
import json
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(__file__).resolve().parent.parent
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
DATA_KEY = "passkey-desired-state.json"
CONFIGMAP_NAME = "keycloak-passkey-desired-state"
NAMESPACE = "keycloak"
# The realm keys this renderer exports, and therefore the ONLY realm keys the
# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
# tuples are identical, so they cannot drift apart silently.
POLICY_STR_FIELDS = (
"webAuthnPolicyPasswordlessRpId",
"webAuthnPolicyPasswordlessRpEntityName",
"webAuthnPolicyPasswordlessRequireResidentKey",
"webAuthnPolicyPasswordlessUserVerificationRequirement",
"webAuthnPolicyPasswordlessAuthenticatorAttachment",
"webAuthnPolicyPasswordlessAttestationConveyancePreference",
)
POLICY_LIST_FIELDS = (
"webAuthnPolicyPasswordlessSignatureAlgorithms",
"webAuthnPolicyPasswordlessAcceptableAaguids",
)
POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+ POLICY_BOOL_FIELDS)
# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
# (DefaultExportImportManager.updateRealm: the policy is applied only when
# RpEntityName is non-null and non-empty). A desired state without it would produce
# an accepted PUT that changes nothing, so refuse to render one.
RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
# The execution whose presence is what makes passkey enrolment necessary.
PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
ENROL_ACTION = "webauthn-register-passwordless"
BIND_KEY = "bind-browser-flow"
BIND_VALUES = ("true", "false")
def committed_bind(root: Path) -> str:
"""The operator's bind-browser-flow value from the COMMITTED file, or "false".
This is the only input read from the output file, and it is validated, not trusted:
anything other than the exact strings "true" / "false" is a hard failure."""
target = root / OUT_FILE
if not target.is_file():
return "false"
docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
data = cm.get("data") or {}
if BIND_KEY not in data:
return "false"
v = data[BIND_KEY]
if not isinstance(v, str) or v not in BIND_VALUES:
raise SystemExit(
"FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
"\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
"at apply time and which would fail the whole platform-config Kustomization"
% (OUT_FILE, BIND_KEY, v))
return v
def render(root: Path) -> str:
"""Return the full text of passkey-desired-state.yaml as it should be committed."""
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
if imp is None:
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
realm = (imp.get("spec") or {}).get("realm") or {}
realm_name = realm.get("realm")
if not isinstance(realm_name, str) or not realm_name:
raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
# ---- the passwordless policy, copied VERBATIM ----------------------------------
policy: dict[str, object] = {}
for f in POLICY_STR_FIELDS:
if f not in realm:
continue # not declared => NOT MANAGED
v = realm[f]
# A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
# a YAML quoting mistake that must not be silently stringified.
if not isinstance(v, str):
raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
"quote it in the realm import" % (IMPORT_FILE, f, v))
policy[f] = v
for f in POLICY_LIST_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
raise SystemExit("FATAL: %s realm field %s is not a list of strings"
% (IMPORT_FILE, f))
policy[f] = list(v) # verbatim, order as authored
for f in POLICY_INT_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, int) or isinstance(v, bool):
raise SystemExit("FATAL: %s realm field %s is not an integer"
% (IMPORT_FILE, f))
policy[f] = v
for f in POLICY_BOOL_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, bool):
raise SystemExit("FATAL: %s realm field %s is not a boolean"
% (IMPORT_FILE, f))
policy[f] = v
if not policy:
# Fail closed. An empty policy would make the reconciler believe every field is
# unmanaged and silently stop reconciling — the realm import's own fail-open.
raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
"refusing to render an empty desired state" % IMPORT_FILE)
if not policy.get(RP_ENTITY_NAME):
raise SystemExit(
"FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
"passwordless policy ONLY when that field is set, so every realm PUT would "
"be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
# ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
browser_flow = realm.get("browserFlow")
if not isinstance(browser_flow, str) or not browser_flow:
raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
"be perfect and still not be the one anyone logs in through"
% IMPORT_FILE)
raw_flows = realm.get("authenticationFlows") or []
if not isinstance(raw_flows, list) or not raw_flows:
raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
flows: dict[str, object] = {}
for fl in raw_flows:
alias = fl.get("alias")
if not isinstance(alias, str) or not alias:
raise SystemExit("FATAL: %s has an authentication flow with no alias"
% IMPORT_FILE)
if alias in flows:
raise SystemExit("FATAL: %s declares the flow %r twice"
% (IMPORT_FILE, alias))
execs = fl.get("authenticationExecutions") or []
if not isinstance(execs, list) or not execs:
raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
% (IMPORT_FILE, alias))
ordered = []
for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
is_flow = bool(ex.get("authenticatorFlow"))
name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
if not isinstance(name, str) or not name:
raise SystemExit(
"FATAL: %s flow %r has an execution with neither an authenticator "
"nor a flowAlias" % (IMPORT_FILE, alias))
requirement = ex.get("requirement")
if not isinstance(requirement, str) or not requirement:
raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
% (IMPORT_FILE, alias, name))
entry: dict[str, object] = {
"kind": "flow" if is_flow else "authenticator",
"name": name,
"requirement": requirement,
}
cfg = ex.get("authenticatorConfig")
if cfg is not None:
if not isinstance(cfg, str) or not cfg:
raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
"authenticatorConfig" % (IMPORT_FILE, alias, name))
entry["configAlias"] = cfg
ordered.append(entry)
flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
if browser_flow not in flows:
raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
% (IMPORT_FILE, browser_flow))
# Every referenced subflow must be declared, or the reconciler's flattening walks
# into a hole and the audit silently covers less than it claims.
for alias, spec in flows.items():
for ex in spec["executions"]:
if ex["kind"] == "flow" and ex["name"] not in flows:
raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
"declared" % (IMPORT_FILE, alias, ex["name"]))
# ---- authenticator configs, copied verbatim ------------------------------------
configs: dict[str, dict] = {}
for cfg in realm.get("authenticatorConfig") or []:
alias = cfg.get("alias")
if not isinstance(alias, str) or not alias:
raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
% IMPORT_FILE)
body = cfg.get("config") or {}
if not isinstance(body, dict) or not body:
raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
% (IMPORT_FILE, alias))
# Values are stringified because Keycloak stores authenticator config values as
# strings. `negate: "false"` in the import and a stored "false" are the same
# declaration, and a YAML-unquoted `false` must not become a Python bool that
# then never matches.
configs[alias] = {k: str(v) for k, v in sorted(body.items())}
# Every configAlias a flow references must be defined, else the reconciler reports
# "git declares config X but defines no key/value for it" on every run forever.
for alias, spec in flows.items():
for ex in spec["executions"]:
ca = ex.get("configAlias")
if ca and ca not in configs:
raise SystemExit(
"FATAL: %s flow %r execution %r references authenticatorConfig %r, "
"which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
# ---- the enrolment required action, DERIVED (see the header) -------------------
offers_passkey = any(
ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
for spec in flows.values() for ex in spec["executions"])
if not offers_passkey:
raise SystemExit(
"FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
"in any flow. The policy decides what a passkey must BE; without that "
"execution nothing ever OFFERS one, so the policy is unreachable "
"configuration and this desired state would be meaningless"
% (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
want = {
"realm": realm_name,
"webAuthnPolicyPasswordless": policy,
# enabled: passkey enrolment must be possible, since the flow offers passkeys.
# defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
# enrol a passkey at next login — the platform-wide lock-out the realm import's
# "additive and optional" posture exists to avoid. Derived, not copied; see the
# header for why the import cannot declare this itself.
"requiredAction": {"alias": ENROL_ACTION, "enabled": True,
"defaultAction": False},
"browserFlow": browser_flow,
"flows": flows,
"authenticatorConfig": configs,
}
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+ " %s: |\n" % DATA_KEY + indented + "\n")
HEADER = """---
# =============================================================================
# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
#
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
# (runs this renderer with --check on every merge)
#
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
# the WebAuthn passwordless policy, the enrolment required action, and the declared
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
# it and (a) converges the policy and the required action onto it, (b) builds the
# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
# writing the flow users are logging in through, and (c) BINDS that verified flow as
# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
#
# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
# operator decision, carried through by the renderer from this committed file (see
# ci/render-keycloak-passkey-policy.py). It defaults to "false".
#
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
# once Done=True, so a realm-level field or a flow edited in git never reaches the
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
# the level-triggered loop that closes that; this file is what it reconciles
# TOWARDS.
#
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
# user-verification requirement and the flow tree would be a second source of truth,
# free to disagree with the realm import — the same "two declarations that can
# disagree" shape as the bug being closed. It is rendered from the import and the
# render is gated, so the two cannot drift.
#
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
# substitution token exactly as the realm import writes it. Both files are rendered
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
# same value in both and the domain is decided in exactly one place.
#
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
# nine more (a hard lock-out of every platform-admin). It is derived instead from
# the presence of a webauthn-authenticator-passwordless execution in the flow:
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
# false, so enrolment stays opt-in.
#
# A field NOT declared on the realm in the import is simply ABSENT here, and the
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: %s
namespace: %s
labels:
app.kubernetes.io/name: %s
app.kubernetes.io/part-of: keycloak
data:
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
BIND_BLOCK = """ # ---------------------------------------------------------------------------
# THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
# "false", QUOTED.
#
# "false" the reconciler builds and verifies the candidate browser flow and
# logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
# Whatever the realm is bound to -- including a rollback -- is left alone.
# "true" it binds the verified candidate in one atomic PUT, and keeps the
# realm bound to it: a binding changed by hand is changed back.
#
# GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
# "Passkeys"), then a PR setting this to "true".
# ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
# one write setting browserFlow back to the previous alias from the Job log.
# ---------------------------------------------------------------------------
%s: "%s"
"""
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
g = ap.add_mutually_exclusive_group()
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
g.add_argument("--check", action="store_true",
help="exit 1 if the committed file has drifted from the source")
args = ap.parse_args()
want = render(ROOT)
target = ROOT / OUT_FILE
if args.write:
target.write_text(want, encoding="utf-8")
print("wrote %s" % OUT_FILE)
return 0
if args.check:
if not target.is_file():
print("CHECK FAIL: %s does not exist. Run: python3 "
"ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
file=sys.stderr)
return 1
have = target.read_text(encoding="utf-8")
if have == want:
print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
return 0
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
"realm on a passkey policy the realm import does not declare, and build "
"the login flow from a tree nobody authored.\n"
"Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
fromfile="committed " + OUT_FILE,
tofile="rendered from " + IMPORT_FILE):
sys.stderr.write(line)
return 1
sys.stdout.write(want)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -445,8 +445,21 @@ join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
check(join is not None,
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
"— without a join target the servers are separate single-node clusters")
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
K3S_TEXT, re.S)
# LINE-BOUNDED ON PURPOSE -- DO NOT "SIMPLIFY" BACK TO `.*` + re.S.
# This was `--server https://.*?\n(?:.*\n){0,10}?\s*when:...` with re.S, which made every `.`
# match newlines too. That is nested unbounded repetition (a `{0,10}` of a quantifier that can each
# swallow the whole document), so a FAILED or late match backtracks exponentially. It cost
# **19m54s of CPU in CI** (measured 2026-10-18 via `docker top` on the live job container) and made
# the whole merge gate take ~28 minutes instead of ~2 on any change touching `infra/**` -- because
# `ci/touched-requirements.sh` maps `infra/*` to R-ENV-01, which pulls in this whole-repo check.
# It was latent for as long as the regex existed and only crossed the threshold when
# roles/k3s_server grew by ~100 lines (the kubelet graceful-shutdown config, PR #71): K3S_TEXT is
# every *.yml in that role concatenated, so the role growing is the input growing.
# `[^\n]*` cannot cross a line, so the `{0,10}` is a genuine ten-line window with nothing to
# backtrack into. Measured on the same input: 0.2 ms, same verdict (match found).
join_when = re.search(
r"--server https://[^\n]*\n(?:[^\n]*\n){0,10}?[ \t]*when:[ \t]*k3s_server_mode[ \t]*==[ \t]*'secondary'",
K3S_TEXT)
check(join_when is not None,
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
"would also try to join itself")
@@ -0,0 +1,108 @@
# =============================================================================
# L0 STATIC — the passkey configuration is derived from one declaration, continuously
# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
# Measured on prod 2026-10-18, hours after the passkey declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
# Done=True Started=False HasErrors=False
#
# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
# them changed nothing about how anyone logs in, and nothing anywhere said so.
# Declarative config that silently does not apply is not declarative config.
#
# GIVEN the repository as authored,
# THEN the reconciler's desired state is GENERATED from the realm import (so the RP ID —
# which is baked into every passkey at enrolment and can NEVER be re-keyed — is
# declared exactly once); both files are rendered by the SAME Flux Kustomization, so
# their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
# policy keys and the reconciler's writable keys are the SAME tuple; the realm
# import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
# delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
# enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
# no ServiceAccount token, mounts only what it needs, passes no mode flag, and
# contains no dollar sign; the endpoint allow-list contains NO delete, NO
# single-execution or config-edit write, and NO required-action write for any alias
# other than the one named literally; the protected-subtree guard exists, is wired
# into the single request choke point and is fail-closed; the SIBLING client-secret
# reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
# (extracted from the shipped ConfigMap and executed) proves compare-then-write,
# idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
# its sub-flows unwritable, the git bind switch and a rollback that sticks, the
# conditional-OTP regression and fail-closed — with mutations.py proving that
# selftest can FAIL.
#
# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
# than only argued: converging a five-flow, ten-execution restructure needs
# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
# calls with no transaction. Doing that to the BOUND flow means a run that dies between
# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
# which is verified by readback before anything is bound. Binding it is then ONE atomic
# PUT of browserFlow, made only when passkey-desired-state.yaml sets
# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
# a schedule tick, and with the switch off an operator's rollback is never undone.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
# Test wraps the SAME script.
#
# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
# password form (needs a real browser against a real realm); that the five flow-building
# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: keycloak-passkey-reconciler
labels:
conformance.home.k3s/R-IDP-03: "true"
conformance.home.k3s/R-IDP-04: "true"
annotations:
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
# check declares it: this invariant spans platform/ (the realm import, the reconciler,
# the generated desired state), platform-config/ (which Flux Kustomization renders
# them, and therefore which substitution pass they share), ci/ (the renderer) and
# clusters/ (that both are reachable from both entrypoints). A change that breaks it
# need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
# then filter this check out of exactly the change that introduces the bug — which is
# how 12 unusable prod OIDC clients sat behind an entirely green cluster on
# 2026-09-06.
conformance.home.k3s/scope: "whole-repo"
# Not safety-critical in the harness's sense (it mutates nothing and needs no
# cluster), but it guards a component that holds WRITE access to the identity
# provider's realm settings.
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The realm's passkey configuration — policy, enrolment action AND browser login
flow — is declared once in the realm import, derived from it by a gated renderer,
and continuously reconciled; the flow converges blue/green, so a one-shot import
that never re-runs can neither leave the configuration inert nor let a reconciler
half-rewrite the login path of the whole platform.
spec:
steps:
- name: audit-passkey-declaration-renderer-and-reconciler
description: >-
Assert the desired state is derived from the realm import and shares its
substitution pass, that the reconciler cannot write the bound flow or delete
anything and binds only on git's bind-browser-flow, that the sibling reconciler
is unwidened, and run the reconciler's own selftest plus the mutation suite that
proves it can fail.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,754 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
# — spec docs/15-identity-and-access.md)
# R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
# — spec docs/15-identity-and-access.md)
#
# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
# ---------------------------------------------------------------
# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
# hours after that declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
# Done=True Started=False HasErrors=False
#
# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
# re-runs it, so every realm-level field and every flow declared after the FIRST
# import is inert — in git, applied by Flux, reported Ready, reaching the realm
# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
# the same fix: a level-triggered reconciler. This check is the static half.
#
# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
#
# GIVEN the repository as authored,
# THEN (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
# (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
# import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
# renderer with --check, so a hand edit that makes the two disagree is
# unmergeable. The RP ID is the sharpest case: it is baked into every passkey
# at enrolment and can NEVER be re-keyed, so two declarations that can skew
# about the domain is an unrecoverable-credential bug, not a cosmetic one.
# (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
# postBuild.substituteFrom is per-Kustomization, so a split would let the
# ${BASE_DOMAIN} token in the realm import and the one in the desired state
# resolve to two different values — and an RP ID mismatch invalidates every
# enrolled passkey with no error the user can act on.
# (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
# declares no realm key outside the reconciler's write surface.
# (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
# reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
# could emit a key the reconciler refuses (fail-closed forever) or the
# reconciler could accept a key the renderer never emits (dead surface).
# (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
# entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
# ServiceAccount token, and contains no dollar sign (it ships inside a
# SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
# rewritten before any container runs).
# (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
# SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
# alias, verify, flip browserFlow in one atomic PUT), so flow-create and
# execution-add endpoints ARE on the allow-list and their presence is not the
# hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
# is the rollback target); the execution- and config-id-scoped write endpoints
# stay off the list because they carry no flow alias and so cannot be screened;
# no required-action write reaches any alias but the one named literally; and
# the guard the allow-list CANNOT express exists in the source and is wired
# into the single request choke point, FAIL-CLOSED until the live binding has
# been read. That last one is what the whole design rests on: a reconciler that
# half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
# this repo has had one (2026-09-06).
# (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
# bind-browser-flow as exactly "true" or "false", the script reads it from the
# SAME mount, and the CronJob passes no flag that could widen it. With "false"
# (the default) the reconciler builds and verifies but never PUTs browserFlow,
# so an operator's rollback sticks; the selftest proves that end to end.
# (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
# allow-list is NOT widened by this work. Its --selftest asserts it refuses
# /users, /groups, /roles, /authentication and the whole-realm endpoint, and
# its guarantee that prod's real user and the media-users / platform-admins /
# viewers groups survive every run must stay LITERALLY true.
# (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
# script from the ConfigMap and executes its `--selftest`, which drives the
# real code paths against an in-process stub Keycloak that MODELS the three
# Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
# passwordless policy, the all-or-nothing policy rebuild that defaults every
# omitted key, and the required-action representation replace.
# (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
# one at a time and requires --selftest to CATCH every one. A passing suite
# proves nothing if it cannot fail, and "reported success while the realm was
# broken" is the exact failure this component exists to end.
# AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
# `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
# via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
# ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
# An unreferenced test is precisely the "looks applied and isn't" failure this
# repo keeps hitting.
#
# NOT proven here (named honestly, in the style of the sibling L0 checks):
# - that the LIVE realm's policy, enrolment action or browser flow match git. That is
# cluster state. The reconciler's own `--audit` mode is the read-only probe for it
# and is what an L2 test should run; no such L2 test is added by this change, so the
# live realm is UNVERIFIED by CI.
# - that the declared browser flow actually WORKS as a login path. Nothing offline can
# prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
# password form, or that a platform-admin is still asked for TOTP. Those need a real
# browser against a real realm.
# - that the reconciler's first real run against prod will succeed. The flow is built
# through five admin-API endpoints whose request shapes were read from the 26.0.6
# source but never exercised against a live Keycloak.
# - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
# offline; the plaintext is not, and must not be.
# - that the CronJob's image can be pulled or its schedule fires.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
RENDERER="ci/render-keycloak-passkey-policy.py"
SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
# Honest SKIP (never green) if the thing under test is absent entirely.
for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
if [ ! -f "${ROOT}/${f}" ]; then
echo "check SKIPPED: ${f} absent — nothing to audit" >&2
exit 3
fi
done
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit of the Keycloak passkey declaration, the renderer that derives the
reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
is one definition of "what a Flux path renders" in the harness."""
from __future__ import annotations
import ast
import json
import re
import subprocess
import sys
import tempfile
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
# test directory is named relative to the repo root the wrapper passes in.
DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
/ "keycloak-passkey-reconciler")
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
import status as st # noqa: E402
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
RENDERER = "ci/render-keycloak-passkey-policy.py"
SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
STATE_CM_NAME = "keycloak-passkey-desired-state"
STATE_DATA_KEY = "passkey-desired-state.json"
BIND_KEY = "bind-browser-flow"
SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
ENROL_ACTION = "webauthn-register-passwordless"
DOLLAR = chr(36)
failures: list[str] = []
def check(cond, msg):
if not cond:
failures.append(msg)
def load(rel):
return list(yaml.safe_load_all((ROOT / rel).read_text()))
# ================================================== (1) the desired state is DERIVED
proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
capture_output=True, text=True, timeout=120)
check(proc.returncode == 0,
f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
f".py --check failed). The reconciler would converge the realm on a passkey policy "
f"the realm import does not declare, and audit the login flow against a tree "
f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
f"domain is an unrecoverable-credential bug.\n "
+ "\n ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
# The realm import must actually declare the things the renderer derives from. If it
# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
# renderer would otherwise be the only signal.
imp_docs = [d for d in load(IMPORT_FILE) if d]
imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
realm = ((imp or {}).get("spec") or {}).get("realm") or {}
policy_keys_in_import = sorted(k for k in realm
if k.startswith("webAuthnPolicyPasswordless"))
check(policy_keys_in_import,
f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
f"passkey policy for the reconciler to converge")
check(realm.get("browserFlow"),
f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
f"BOUND changes nothing about how anyone logs in")
# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
# representation as REPLACE, not merge, so declaring ONE action imports exactly that
# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
# NOT ADD ONE" is advice; this is the gate.
check("requiredActions" not in realm,
f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
f"as REPLACE, not merge: declaring any action imports exactly those plus "
f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
f"realm's browser flow still demands a TOTP they could no longer enrol. The "
f"enrolment action is already registered by default (priority 80, enabled, "
f"defaultAction false) and the passkey reconciler converges it through the "
f"per-alias admin API, which has no such hazard")
# ============================== (2)(3) the two files share one substitution pass
state_docs = [d for d in load(STATE_FILE) if d]
state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
desired = {}
if state_cm is not None:
check(state_cm["metadata"]["name"] == STATE_CM_NAME,
f"{STATE_FILE}: the ConfigMap is named "
f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
data = state_cm.get("data") or {}
check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
f"PASSKEY_BIND_FILE defaults would not resolve")
# THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
# at apply time -- failing the whole platform-config Kustomization, realm import
# included. The renderer refuses it too; asserted here so the reason is on record.
check(data.get(BIND_KEY) in ("true", "false"),
f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
f"string \"true\" or \"false\"")
raw = data.get(STATE_DATA_KEY, "")
try:
desired = json.loads(raw)
except ValueError as e:
check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
f"({e}) — the reconciler would fail closed on every run")
# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
import_owners = set()
state_owners = set()
recon_owners = set()
for env in ("dev", "prod"):
entry = f"clusters/{env}"
if not (ROOT / entry).is_dir():
continue
applied, _ = st.reconciled_files(ROOT, entry)
check(RECON_FILE in applied,
f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
f"healer would exist on one cluster only")
check(STATE_FILE in applied,
f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
f"would fail closed on that cluster forever")
check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
# listed as resources of the same kustomization.yaml.
for kpath in ROOT.glob("*/kustomization.yaml"):
try:
k = yaml.safe_load(kpath.read_text()) or {}
except Exception:
continue
res = [str(r) for r in (k.get("resources") or [])]
base = kpath.parent.name
if any(r.endswith("keycloak-realm-import.yaml") for r in res):
import_owners.add(base)
if any(r.endswith("passkey-desired-state.yaml") for r in res):
state_owners.add(base)
if any(r.endswith("passkey-reconciler.yaml") for r in res):
recon_owners.add(base)
check(import_owners and import_owners == state_owners,
f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
check(recon_owners == state_owners,
f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
f"{sorted(state_owners)} — the reconciler could be applied without its input")
# (3) no substitution token the realm definition does not also use.
import_text = (ROOT / IMPORT_FILE).read_text()
TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
import_tokens = set(TOKEN.findall(import_text))
# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
# policy uses, so a subset check is the right assertion rather than an exact match.
check(state_tokens <= import_tokens,
f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
f"silently — which for an RP ID means every passkey ceremony fails")
# ======================================= (4) the two write surfaces are the SAME tuple
recon_docs = [d for d in load(RECON_FILE) if d]
cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
script = ""
if cm is not None:
data = cm.get("data") or {}
check(len(data) == 1,
f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
f"(found {sorted(data)}) — a second copy would be untested by this gate")
check(SCRIPT_KEY in data,
f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
script = next(iter(data.values()), "")
check(len(script.splitlines()) > 200,
f"{RECON_FILE}: the extracted script is implausibly short "
f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
f"selftest below would prove nothing")
# THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
# postBuild.substituteFrom, so envsubst rewrites it before any container runs.
check(DOLLAR not in script,
f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
f"never needs one; chr(36) spells it where a literal is genuinely required")
def tuple_literal(src, name):
"""The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
than importing it keeps this check offline and side-effect free, and means the thing
asserted is the text that ships."""
try:
tree = ast.parse(src)
except SyntaxError:
return None
for node in tree.body:
if isinstance(node, ast.Assign):
for t in node.targets:
if isinstance(t, ast.Name) and t.id == name:
try:
return ast.literal_eval(node.value)
except ValueError:
return None
return None
renderer_src = (ROOT / RENDERER).read_text()
# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
# their parts from the same vocabulary; a part present in one and not the other simply
# contributes nothing, which is why the comparison below is of the resolved SETS.
def resolve_policy(src):
parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
"POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
"POLICY_BOOL_FIELDS")
out = []
for p in parts:
v = tuple_literal(src, p)
if v:
out.extend(v)
return sorted(set(out))
recon_keys = resolve_policy(script)
rend_keys = resolve_policy(renderer_src)
check(recon_keys and recon_keys == rend_keys,
f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
f"every run fails closed forever; if the reconciler accepts a key the renderer "
f"never emits, that is dead write surface on the identity provider")
# Every exported key must actually be a passwordless key. The two-factor
# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
# with no effect.
check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
f"the reconciler's writable keys include a non-passwordless field: "
f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
f"writing it is a silent no-op")
# And the desired state must declare only those.
if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
check(not extra,
f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
f"surface — it would fail closed on every run")
# The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
check(desired["webAuthnPolicyPasswordless"].get(
"webAuthnPolicyPasswordlessRpEntityName"),
f"{STATE_FILE} carries no non-empty "
f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
f"passwordless policy only when that field is set "
f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
f"accepted with 204 and change nothing")
# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
# import's stated "additive and optional" decision. It is one boolean, so it is gated.
ra = desired.get("requiredAction") or {}
check(ra.get("alias") == ENROL_ACTION,
f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
f"any other value is unreachable and the pass would fail closed")
check(ra.get("enabled") is True,
f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
f"impossible while the login flow offers passkeys")
check(ra.get("defaultAction") is False,
f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
f"must be false. True forces EVERY user to enrol a passkey at next login — this "
f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
# ========================================= (5) the reconciler manifest is safe + wired
if cj is not None:
pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
ctr = pod["containers"][0]
vols = {v["name"]: v for v in pod.get("volumes", [])}
mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
env = {e["name"]: e for e in ctr.get("env", [])}
# NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
# narrower than reading the KeycloakRealmImport CR through the API: no Role, no
# RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
check(pod.get("automountServiceAccountToken") is False,
f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
f"ServiceAccount token is pure blast radius (R-SEC-02)")
check("serviceAccountName" not in pod,
f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
check(STATE_CM_NAME in names,
f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
check(cm["metadata"]["name"] in names,
f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
# NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
# secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
check(not [v for v in vols.values() if "secret" in v],
f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
f"credential is referenced by secretKeyRef only; it must never be projected "
f"as a file here")
# The mount paths must be the ones the code DEFAULTS to, or the container starts and
# fails closed forever while every manifest looks correct.
state_vol = next((n for n, v in cm_vols.items()
if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
if state_vol:
mp = (mounts.get(state_vol) or {}).get("mountPath")
check(mp == "/etc/keycloak-passkey",
f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
f"script's PASSKEY_STATE_FILE default is "
f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
f"the mount path — they are a pair and must move together")
check((mounts.get(state_vol) or {}).get("readOnly") is True,
f"{RECON_FILE}: the desired-state mount must be readOnly")
check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
f"(safe, but git's \"true\" would then silently never take effect)")
check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
f"keys may not be projected")
# The bind decision belongs to git. A container arg could only NARROW it today
# (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
# behaviour must be readable from passkey-desired-state.yaml alone.
cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
check(not [x for x in cmd if x.startswith("--")],
f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
# Retries burn the bounded candidate-alias budget on a deterministic failure.
check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
f"within a tick, and each retried build consumes another candidate alias")
# The admin credential, referenced BY NAME. Never decrypted, never a literal.
for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
e = env.get(key) or {}
ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
check(ref.get("name") == "keycloak-admin-credentials",
f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
f"Secret by secretKeyRef, got {e!r}")
check("value" not in e,
f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
f"manifest is a plaintext secret in git")
check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
f"{RECON_FILE}: KEYCLOAK_REALM is "
f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
f"wrong realm, or none")
check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
f"{RECON_FILE}: readOnlyRootFilesystem must be true")
check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
f"{RECON_FILE}: allowPrivilegeEscalation must be false")
check(pod["securityContext"].get("runAsNonRoot") is True,
f"{RECON_FILE}: runAsNonRoot must be true")
check(cj["spec"].get("concurrencyPolicy") == "Forbid",
f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
f"would double the write attempts against the IdP during an outage recovery")
check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
#
# This component converges the browser flow BLUE/GREEN: it builds the target structure
# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
# first design of this check the mere presence of a flow write is no longer the thing to
# forbid. What must hold instead is narrower and stronger:
#
# (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
# (b) the single-execution and config-edit endpoints stay off the list — requirements
# are set through the flow-scoped endpoint, which the guard in (d) screens;
# (c) no required-action write may reach any alias but the one named LITERALLY;
# (d) and the guard the allow-list CANNOT express must exist in the source: a
# method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
# flow's subtree, FAIL-CLOSED until that binding has been read.
#
# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
# change (2026-09-06).
allow = tuple_literal(script, "_ALLOWED") or ()
check(bool(allow),
f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
f"script. This check's central assertions about the write surface cannot be made, "
f"so it must FAIL rather than pass silently")
if allow:
check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
f"allow-list cannot distinguish reading a flow from writing one, which this "
f"component's whole safety argument rests on")
writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
# (a) nothing is ever deleted.
check(not [(m, p) for m, p in allow if m == "DELETE"],
f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
f"does requires deleting anything: a superseded flow is left unbound as the "
f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
# (b) the endpoints that would let it edit live objects in place.
banned = [(m, p) for m, p in writes
if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
r"\[\^/\]\+", p)
or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
p)
or p.endswith("/authentication/register-required-action")]
check(not banned,
f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
f"and config-edit endpoints must stay off it: requirements are set through the "
f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
f"therefore screenable against the bound subtree. An execution- or "
f"config-id-scoped write is not, so it would be an unscreened hole")
# (c) no required-action write may reach any other alias.
ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
f"permanent lock-out of every platform-admin, because auth-otp-form in the "
f"browser flow still demands a TOTP that could no longer be enrolled")
check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
f"must be named LITERALLY so no other action is writable by construction")
# Nothing outside the realm + authentication surface at all.
stray = [(m, p) for m, p in allow
if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
check(not stray,
f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
f"be unreachable (prod's real user and the media-users / platform-admins / "
f"viewers groups must survive every run); clients are the SIBLING "
f"reconciler's surface and the two must not overlap")
# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
# Asserted against the SOURCE, because this is the property the blue/green design rests
# on and a comment claiming it is worth nothing. Three things must all be true.
check(re.search(r"^def _screen_write\(", script, re.M) is not None,
f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
f"one thing the allow-list cannot express, because its write entries are "
f"alias-parameterised. Without it this component can mutate the live login path")
# It must be CALLED from the single choke point, not merely defined.
check(re.search(r"_screen_write\(method, path", script) is not None,
f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
f"request's method and path. A guard that is not on the request path is decoration")
# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
# the top of every run must put them back there.
reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
check(reset_fn is not None
and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
f"must start each run UNDETERMINED, because while they are, every flow write is "
f"refused — that is the fail-closed half of the guard, and it is what stops a "
f"build running before the live binding has been read")
check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
f"protected sets are not reset per run and a previous run's binding could be "
f"trusted for this one")
# The create-flow endpoint carries NO alias in its path, so it must be screened on its
# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
# future edit cannot silently undo it.)
check("_FLOW_CREATE_PATH" in script,
f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
f"That endpoint carries no alias in its path — the alias is in the request BODY — "
f"so without a body-based screen it bypasses the protected-subtree guard "
f"altogether")
# And the one function allowed to change the live binding must be distinct from the
# policy write, which must refuse to.
# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
# the top-level flow alone while every selftest passed against a stub that returned it.
check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*? name = ex\.get\("displayName"\)',
script, re.M) is not None,
f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
f"None and the protected subtree silently shrinks to the top-level flow")
# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
f"and ignores the path alias, so a path-only screen lets it write the bound flow")
check("def declared_bind(" in script and "BIND_FILE" in script,
f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
check("def bind_browser_flow(" in script,
f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
f"one named, auditable operation, not an incidental field in some other write")
check(re.search(r"never rebinds the browser flow", script) is not None,
f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
f"can rebind the login path by accident — to an UNVERIFIED flow")
# ======================================== (7) the SIBLING reconciler is NOT widened
if (ROOT / SIBLING_FILE).is_file():
sib_docs = [d for d in load(SIBLING_FILE) if d]
sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
check(bool(sib_allow),
f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
f"prove the sibling was not widened")
if sib_allow:
# The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
# whole-realm endpoint and the /authentication tree; assert the same here so a
# future edit that "just adds the realm endpoint there instead" is caught.
bad = [p for p in sib_allow
if isinstance(p, str)
and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
or "authentication" in p or "/users" in p or "/groups" in p
or "/roles" in p)]
check(not bad,
f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
f"stated guarantee — prod's real user and the media-users / "
f"platform-admins / viewers groups survive every run — is expressed as "
f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
f"The passkey work is a SIBLING program with its own narrow list precisely "
f"so that guarantee stays literally true. Revert the widening")
# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
if script:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "reconcile.py"
p.write_text(script)
proc = subprocess.run([sys.executable, str(p), "--selftest"],
capture_output=True, text=True, timeout=300)
tail = (proc.stdout + proc.stderr).strip().splitlines()
check(proc.returncode == 0,
"the passkey reconciler's --selftest FAILED (compare-then-write / "
"idempotence / strict read-modify-write / the RpEntityName gate / "
"flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
" " + "\n ".join(tail[-14:]))
if proc.returncode == 0:
print(" passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
# ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
# A passing suite proves nothing unless it can be shown to FAIL. mutations.py
# injects plausible regressions one at a time -- a partial policy PUT, a
# rebind of browserFlow, a widened allow-list that makes the flow writable, a
# fail-open on an unreadable input -- and requires --selftest to catch every
# one. This matters more than usual here: the whole incident class is a
# component that REPORTED SUCCESS while the realm was broken, and a green test
# that cannot fail is the same lie one level up.
mut = DIR_PATH / "mutations.py"
if mut.is_file():
mproc = subprocess.run([sys.executable, str(mut), str(p)],
capture_output=True, text=True, timeout=900)
mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
check(mproc.returncode == 0,
"the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
"one injected regression survived it, so the suite does not prove "
"what it claims:\n " + "\n ".join(mtail[-16:]))
if mproc.returncode == 0:
print(" passkey reconciler mutations: "
+ (mtail[-1] if mtail else "(no output)"))
else:
check(False,
"conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
"is missing — nothing proves the --selftest can fail, and an "
"unfalsifiable proof is the failure mode this component exists to end")
# ======================================================= (9) this check is load-bearing
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
is not None,
"keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
"only when a change happens to attribute one of the requirements it is labelled "
"with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
"change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
"prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
"stage is a purely requirement-scoped run again and the scope annotation is inert "
"decoration — this check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
"ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
"import or the reconciler would attribute NO requirement and the keycloak tests "
"would be filtered out of the merge gate's touched subset")
# =============================================================================
if failures:
print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
for f in failures:
print(" FAIL %s" % f, file=sys.stderr)
raise SystemExit(1)
print("keycloak-passkey-reconciler: all assertions passed "
"(desired state derived from the realm import, one substitution pass, write "
"surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
"deletable, sibling allow-list unwidened, selftest green and falsifiable)")
PY
@@ -0,0 +1,456 @@
#!/usr/bin/env python3
# =============================================================================
# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
#
# WHY THIS EXISTS
# ---------------
# A selftest that passes proves nothing on its own: a selftest asserting `True ==
# True` also passes. This reconciler exists because a one-shot realm import
# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
# with the whole passkey block inert, measured on prod 2026-10-18), and the same
# trap applies one level up -- a green test suite that cannot fail is the same lie
# in a different place.
#
# It matters more here than for the sibling client reconciler, because this program
# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
# untested safety property on that surface is not a documentation gap, it is an
# outage waiting for a schedule tick.
#
# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
# a time, and requires that --selftest FAILS for every one. A mutation that
# SURVIVES means the suite does not actually check that property, and the check
# fails, naming it.
#
# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
# write -- never mutating the flow users are logging in through. Each of those
# properties is enforced in code; these mutations are the proof that the enforcement
# is load-bearing rather than commentary. Several model the obvious "simplification"
# someone will propose:
# * "just build it under the declared alias, the suffix is ugly"
# * "bind it first, then check -- the check is just belt and braces"
# * "the protected-alias screen is redundant, the allow-list covers it"
# Every one of those would be a total SSO outage waiting to happen.
#
# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
# The sibling mutations.py dedents by four because its anchors are copied out of the
# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
# which is one less transformation to get wrong. check.sh passes the extracted file,
# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
#
# python3 mutations.py <path-to-extracted-reconciler.py>
#
# Runtime is ~60s (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster).
# =============================================================================
from __future__ import annotations
import re
import subprocess
import sys
import tempfile
from pathlib import Path
# (name, the mistake being modelled, anchor, replacement)
MUTATIONS: list[tuple[str, str, str, str]] = [
# ---------------------------------------------------------------- blue/green safety
("build INTO the currently bound flow",
"'the suffix is ugly, just build under the declared alias' -- every "
"create-execution and set-requirement call would then land on the flow users are "
"logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
"every SSO app and kubectl. This is the single regression the whole design exists "
"to make impossible",
" def mapped(alias):\n"
" return \"%s-%s\" % (alias, suffix)",
" def mapped(alias):\n return alias"),
("the protected-subtree screen is never called",
"'the allow-list already covers it' -- it does not: the allow-list's write entries "
"are alias-parameterised, so only this screen can say 'but not the flow that is "
"bound'. Without it the reconciler may write anywhere in the authentication tree",
" _screen_write(method, path, body if isinstance(body, dict) else None)",
" pass"),
("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
"reset them to frozenset() so the code reads more simply -- it silently removes "
"the fail-closed property, so a build that ran before the live binding had been "
"read would be PERMITTED, and could write straight into the live flow. (The "
"module-level initial value is deliberately NOT the anchor here: run() resets "
"these at the start of every run, so the initialiser is unreachable and mutating "
"it would prove nothing.)",
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
" _PROTECTED_FLOW_ALIASES = None\n"
" _PROTECTED_EXECUTION_IDS = None",
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
" _PROTECTED_FLOW_ALIASES = frozenset()\n"
" _PROTECTED_EXECUTION_IDS = frozenset()"),
("bind BEFORE verifying the build",
"'bind it first, the readback is just belt and braces' -- it is not: the readback "
"is what catches a build that was accepted but came out wrong, including the "
"conditional-OTP config landing under the wrong key. Binding first makes that the "
"live login path before anyone notices",
" verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
" \"the newly built flow %r\" % target)",
" verify = []"),
("a failed build is reported as SUCCESS",
"swallow the build error and return a converged result -- the realm is then left "
"on the old flow (which blue/green makes safe) but the run says it converged, so "
"nobody ever learns the passkey flow was never built. That is the one-shot "
"import's fail-open, reintroduced. NOTE: simply removing the early return here "
"is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
"is defence in depth working as intended -- so the mutation targets the reporting",
" problems.append(\n"
" \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
" \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
" \"place, unbound and harmless\" % (candidate, e))\n"
" return 0, problems",
" return 1, []"),
("an existing candidate flow is reused WITHOUT verifying it",
"'if the alias exists we already built it' -- a half-built leftover from a crashed "
"run lives under exactly that alias, so this binds a partially constructed login "
"path",
" if cand_shape is not None and not compare_shape(\n"
" cand_shape, expected, base, realm, token, configs,\n"
" \"the existing flow %r\" % candidate):",
" if cand_shape is not None:"),
# ------------------------------------- 2026-10-29 review findings, each made falsifiable
("sub-flows identified by `alias` again",
"read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
"recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
"sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
"a write into browser-mfa-forms would NOT be refused",
" name = ex.get(\"displayName\")\n"
" return name if isinstance(name, str) and name else None",
" name = ex.get(\"alias\")\n"
" return name if isinstance(name, str) and name else None"),
("an unnameable sub-flow is tolerated",
"carry on when a sub-flow row has no displayName -- the protected subtree is then "
"SMALLER than the live tree, which is the fail-open the guard exists to prevent",
" if name is None:\n"
" return None, None",
" if name is None:\n"
" name = \"\""),
("the requirement PUT is screened on its path only",
"trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
"body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
"the path can carry a BOUND execution's id in the body",
" targets.append((\"execution\", (body or {}).get(\"id\")))",
" pass"),
("execution-addressed writes are not held to the build tree",
"'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
"outside it too, so a stray id would edit an unrelated flow",
" if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
" or target not in _BUILD_EXECUTION_IDS):",
" if False:"),
("the requirement PUT may carry any key",
"let the body carry flowId / displayName -- the same handler then RENAMES whatever "
"flow flowId names, bound or not",
" if extra:\n"
" raise Refused(",
" if False:\n"
" raise Refused("),
("git's bind switch is ignored",
"bind whenever verification passes, as the first version did -- the first live use of "
"an untested login flow becomes a schedule tick, and an operator's rollback is "
"undone within 15 minutes, forever",
" bind = git_bind and not no_bind",
" bind = not no_bind"),
("bind=false still binds inside the flow pass",
"keep the flag but forget to honour it where the bind happens",
" if not bind:\n"
" # THE BIND SWITCH IS OFF.",
" if False:\n"
" # THE BIND SWITCH IS OFF."),
("an absent bind switch means bind",
"default to true when the key is missing -- a ConfigMap predating the switch, or a "
"deleted line, would silently put an untested flow live",
" except FileNotFoundError:\n"
" return False",
" except FileNotFoundError:\n"
" return True"),
("an invalid bind switch is guessed at",
"read anything that is not \"true\" as false -- a typo then passes silently instead of "
"stopping the run, and the operator believes the flow is live",
" if raw == \"false\":\n"
" return False\n"
" raise ValueError(",
" return False\n"
" raise ValueError("),
("bind=false rebuilds beside a hand-edited bound flow",
"treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
"off this builds an -r2 next to the operator's hotfix on every tick",
" if candidate in have and candidate == bound and not bind:",
" if False:"),
# ------------------------------------------------------------------- idempotence
("sub-flow aliases are compared as names",
"compare the sub-flow alias like any other field -- the flows this program builds "
"carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
"converged: it would be rebuilt and the login path reflipped every fifteen "
"minutes, forever",
" out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
" out.append((level, ex[\"kind\"], ex[\"name\"],\n"
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
("the converged short-circuit is removed",
"drop the 'bound flow already matches git' early return -- every run rebuilds and "
"reflips, which is a permanent write loop against the IdP and churns the realm's "
"login path on a schedule",
" if not drift:\n"
" log(\"flow ok (the bound flow %r matches git: %d execution(s), no \"\n"
" \"write)\" % (bound, len(expected)))\n"
" return 0, problems",
" if False:\n return 0, problems"),
("the execution priority is dropped from the requirement write",
"omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
"missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
"the execution order while only a requirement change was intended. The passkey "
"branch would end up in the wrong place in the tree",
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
" \"priority\": ex.get(\"priority\") or 0})",
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
("the alias budget is unbounded",
"loop forever looking for a free alias -- a persistently failing build then creates "
"unbounded flow sprawl in the realm, and nothing here can delete any of it",
" for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
" for attempt in range(1, 10000):"),
("fail-open when the bound flow cannot be read",
"carry on when the live flow's executions are unreadable -- the program then cannot "
"know which flows are live, so it cannot prove a build would miss them, and it "
"would build with the protected sets still undetermined",
" return 0, [\n"
" \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
" \"be read, so this program cannot establish which flows are live. Flow \"\n"
" \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
" % bound]",
" _set_protected(set(), set())\n bound_shape, bound_raw = None, None"),
# ------------------------------------------------------- the conditional-OTP regression
("authenticator config contents no longer compared",
"check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
"regression: the original bug wrote the role under "
"`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
"to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
"and every platform-admin got a 403 from every SSO-gated app",
" if str(have_cfg.get(key)) != str(want_cfg[key]):",
" if False:"),
("undeclared config keys are ignored",
"stop reporting keys the realm holds and git does not -- the stale "
"`condition.evaluated.user.role` key is the signature of the original null-role "
"bug, and a flow carrying both keys would look correct",
" extra = sorted(k for k in have_cfg if k not in want_cfg)",
" extra = []"),
("flow structure divergence detected but not reported",
"leave the per-step comparison out -- the realm's login flow could be the old "
"pre-passkey shape forever and every run would report the realm converged, which "
"is the exact fail-open of the one-shot import this program replaces",
" if observed[idx][:4] != expected[idx][:4]:",
" if False:"),
("the browserFlow BINDING is no longer compared",
"stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
"passkey flow that nothing logs in through changes nothing at all",
" if bound_shape is not None:\n"
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
" if False:\n"
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
# ------------------------------------------------------------- realm / policy writes
("partial policy PUT instead of read-modify-write",
"send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
"passwordless policy from the representation and substitutes its own DEFAULT for "
"every key the body omits, so a partial PUT silently resets everything it did not "
"mention while returning 204",
" body = dict(realm_rep)\n"
" for key in drift:\n"
" body[key] = wanted[key]",
" body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
" BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
" for key in drift:\n"
" body[key] = wanted[key]"),
("the policy pass rebinds browserFlow as a side effect",
"'while we are here, let the policy write fix the flow binding too' -- the realm "
"PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
"rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
"the passkey policy",
" if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
" raise RuntimeError(",
" body[BROWSER_FLOW] = \"browser\"\n"
" if False:\n"
" raise RuntimeError("),
("the RpEntityName gate removed from desired-state validation",
"accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
"passwordless policy ONLY when that field is non-empty, so every realm PUT would "
"be accepted with 204 and change nothing: reported success, zero convergence",
" if not policy.get(RP_ENTITY_NAME):\n"
" raise ValueError(",
" if False:\n raise ValueError("),
("the RpEntityName gate removed from the write path",
"drop the second, defence-in-depth guard -- it is unreachable while the "
"desired-state validation holds, which is exactly why deleting it would otherwise "
"be an invisible regression",
" if not body.get(RP_ENTITY_NAME):\n"
" raise RuntimeError(",
" if False:\n raise RuntimeError("),
("no readback verification on the policy write",
"trust the 204 -- a server that accepts the PUT and does not apply it is reported "
"as REPAIRED, which is 'reports success while the realm never converged' again, "
"and it is precisely what the RpEntityName gate does when it is missed",
" still = policy_drift(after, wanted)",
" still = {}"),
("AAGUID comparison made order-sensitive",
"compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
"is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
"realm on every run forever while every run still reports success",
" if stored != sorted(declared_value):\n"
" drift[key] = (stored, sorted(declared_value))",
" if stored != declared_value:\n"
" drift[key] = (stored, declared_value)"),
("signature-algorithm order treated as insignificant",
"sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
"to the authenticator, so ES256-then-RS256 is not the same declaration as "
"RS256-then-ES256, and a reorder in the realm would never be corrected",
" if stored != list(declared_value):",
" if sorted(stored) != sorted(declared_value):"),
("unmanaged policy keys forced to a value",
"iterate every writable key instead of only the declared ones -- a "
"webAuthnPolicyPasswordless field the realm holds and git does not declare would "
"be overwritten, turning 'not declared' into 'must equal my default'",
" for key in sorted(wanted):\n declared_value = wanted[key]",
" for key in POLICY_FIELDS:\n declared_value = wanted.get(key, \"\")"),
# ------------------------------------------------------------------ required actions
("required-action PUT sends only the two drifted fields",
"a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
"the representation, so name, priority and config are all blanked: the enrolment "
"action keeps working but loses its priority and display name",
" body = dict(rep)\n for key in drift:\n body[key] = wanted[key]",
" body = {}\n for key in drift:\n body[key] = wanted[key]"),
("the guarded default required actions are no longer audited",
"stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
"exist -- their disappearance is the signature of a requiredActions block in the "
"realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
"lock-out of every platform-admin",
" missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
" missing = []"),
# ----------------------------------------------------------------- inputs / coupling
("the three passes are coupled",
"abort the run at the first failing pass -- a policy write that fails for an "
"unrelated reason would then suppress the browser-flow pass, which is the most "
"safety-critical thing here",
" problems.extend(found)\n if found:\n rc = 1",
" problems.extend(found)\n if found:\n rc = 1\n"
" break"),
("policy-key validation removed from the desired state",
"accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
"writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
"bruteForceProtected or any other top-level realm field on the live IdP",
" if key not in POLICY_FIELDS:",
" if False:"),
("fail-open on an unreadable desired state",
"treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
"reconciling nothing, which is precisely the fail-open of the realm import this "
"whole component exists to replace",
" if not isinstance(raw, dict) or not raw:\n"
" raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
" if not isinstance(raw, dict) or not raw:\n return {}"),
("an undeclared authenticatorConfig reference is tolerated",
"stop validating that every configAlias a flow references is actually declared -- "
"the build then fails at the config-binding step, AFTER flows and executions have "
"already been created, leaving leftovers on every run",
" if ca and ca not in configs:\n"
" raise ValueError(",
" if False:\n raise ValueError("),
("the flow-cycle guard is removed",
"drop the `seen` check -- a cyclic flow declaration then recurses until the "
"interpreter dies, in a CronJob that retries",
" if alias in seen:\n"
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
" if False:\n"
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
]
def main() -> int:
if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2
src = Path(sys.argv[1]).read_text()
problems: list[str] = []
killed = 0
for name, why, old, new in MUTATIONS:
# An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything.
n = src.count(old)
if n != 1:
problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n))
continue
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(src.replace(old, new, 1))
path = fh.name
try:
proc = subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=300)
finally:
Path(path).unlink(missing_ok=True)
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-54s %s" % (name, how))
print()
if problems:
print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
% (len(problems), len(MUTATIONS)), file=sys.stderr)
for p in problems:
print(" FAIL %s" % p, file=sys.stderr)
return 1
print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
"(the suite is falsifiable, not decorative)" % killed)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,106 @@
---
date: 2026-10-17
branch: feat/graceful-cluster-shutdown
title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
---
## What changed
All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
the whole sequence was improvised live. Two defects came out of doing it.
**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
`shutdown-grace-period-critical-pods=20s`.
The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
period above that cap is truncated with no error, so the feature looks configured and still
hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
relationship, because the failure is invisible.
**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
that PVC was the only copy of its log data. This is the defect in this change set that actually
destroyed something, and it was mine.
New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
`oauth2-proxy` replica).
## What I verified
* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
`logind` or `InhibitDelay` setting exists anywhere in the tree.
* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
`platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
(`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
`200/200/401` for apex/gitea/plex.
* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
`{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
Longhorn volume count had gone 25 → 24.
* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
`2/2 Running`, and the volume count returned to 25.
* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
scale).
* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
function returns the 13 workloads currently holding a PVC.
* `bash -n` clean; the two edited Ansible YAML files parse.
## What I did NOT verify
* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
run the role, did not restart k3s, and have **not** observed a node actually drain on
`shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
terminate gracefully instead of vanishing.
* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
node takes noticeably longer to go down, this is why.
* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
`kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
kind of thing that makes k3s fail to come up. **Review this one before applying.**
* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
defaults, wrong if someone writes a compound duration.
* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
that worked but has not itself been executed end to end. Only `status` and the discovery and guard
functions have run.
* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
one at risk. The runbook gives the one-liner to list them.
* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
offline subset only.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
+311
View File
@@ -0,0 +1,311 @@
---
date: 2026-10-17
branch: feat/keycloak-passkeys
title: Passkeys declared for the homelab realm — and the realm import will not apply them
---
## What changed
`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
login for the `homelab` realm, in two parts:
1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
`requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
`authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
`acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
```
browser-mfa (top level, unchanged)
browser-mfa-forms ALTERNATIVE
auth-username-form REQUIRED <- was auth-username-password-form
browser-mfa-passkey-or-password REQUIRED <- NEW sub-flow
webauthn-authenticator-passwordless ALTERNATIVE <- NEW: the passkey
browser-mfa-password-and-otp ALTERNATIVE <- NEW sub-flow
auth-password-form REQUIRED <- was half of the combined form
browser-mfa-conditional-otp CONDITIONAL (unchanged, relocated intact)
```
The design decisions, with their reasoning, are written into the manifest comments in this repo's
usual density. The ones that matter:
* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
`kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
not price that risk.
* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
"What I verified".
* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
password branch only. The justification is `userVerificationRequirement: required` — the
ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
one-line edit to reverse this decision.
* **Login becomes two screens** (username, then password) for every user, passkey or not, because
the server must know who is logging in before it can decide whether a passkey exists. Accepted
cost; it is the enabling change.
* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
verified" for the measured reason it would have made every login open on a passkey prompt.
`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
this does not reach the live realm, the two operator paths that would apply it, and the four-step
browser verification nobody has run.
No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
## What I verified
**The live realm import has already completed and will never re-run.** Measured on prod,
2026-10-17:
```
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
{"conditions":[{"message":"","status":"True","type":"Done"},
{"message":"","status":"False","type":"Started"},
{"message":"","status":"False","type":"HasErrors"}]}
```
`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
documents for client secrets and client URLs. **So the change in this PR has zero effect on how
anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
policy fields are not client fields.
**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
silently rather than rejected — which is why this was checked rather than assumed.
**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
`26.0.6` tag, not from memory or from docs.
**Every provider ID is quoted from upstream source, not guessed.**
| String I used | Where it is defined at tag 26.0.6 |
|---|---|
| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
```java
if (rep.getRequiredActions() != null) {
for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
DefaultRequiredActions.addDeleteAccountAction(newRealm);
} else {
DefaultRequiredActions.addActions(newRealm);
}
```
Declaring one required action therefore skips the entire default set. `RealmManager` is no
backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
delivered by a change whose purpose was to *add* a login method.
**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
`defaultAction=false` *is* the additive-and-optional posture this change wants.
**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
passkey, no passkey screen, straight to the password form.
**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
the username form is offered to *everyone*, including users who will never own one.
**The policy sentinels are read from the shipped code, and one of my initial assumptions was
wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
the WebAuthn-API-level one. I checked both because they disagree. The login theme's
`webauthnRegister.js` at 26.0.6 settles the behaviour:
```js
if (input.requireResidentKey !== 'not specified') {
if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
else { authenticatorSelection.requireResidentKey = false; }
}
...
if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
rp: {id: input.rpId, name: input.rpEntityName}
```
So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
express `residentKey: "preferred"` at all.
`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
(`DefaultExportImportManager.java:842`) drops the whole policy silently.
**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
back exactly as designed (five flows, the executions and priorities above, `browserFlow:
browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
the 160-column ceiling.
**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
'browser-mfa'`. Renaming the top-level flow would have failed that test.
**Envsubst safety.** `platform-config`'s Flux Kustomization declares
`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
unparseable token cannot fail the whole Kustomization.
**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
```
[ ok ] stage spec-immutability: pass <- NOT skipped; both shim mounts worked, R-TEST-06 covered
[ ok ] stage decisions-log: pass <- archive frozen, 48 entries well-formed
[ ok ] stage lint: pass
[ ok ] stage l0: pass <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
[ ok ] stage render-golden: pass
[ ok ] stage l1: pass
[ ok ] stage approval-policy: pass <- no sensitive path touched, so no non-author review needed
[skip] stage gitea-settings: SKIPPED <- needs SETTINGS_API_TOKEN, CI-only; expected
[skip] stage conformance: SKIPPED <- cluster-gated L2/L3; expected
MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
```
No lint finding and no check failure names either file I changed. Touched requirements resolved to
`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
**A note on how that number was obtained, because the first attempt was worthless.** My first gate
run returned the same rc 3 and the same seven passes — against a tree that did not contain this
change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
the second run, after committing and rebasing.
**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
## What I did NOT verify
**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
admin-console reconstruction of this flow matches what is declared here is, by construction,
unverifiable from git.
**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
exercised by a real WebAuthn ceremony on any device. Specifically unverified:
* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
`keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
**silently, in the browser, before Keycloak is involved**, with no server-side log line. If
enrolment produces no OS prompt and no error, this is the first suspect.
* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
compare, but I have not seen a realm import accept it. Omitting the field would default to the
same value; if an import ever rejects the literal, drop the line.
* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
than rejects it, is reasoning about browser behaviour, not a measurement.
* **Whether a passkey user can still reach the password form.** I argued it from
`createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
user holding both credentials the *first* screen is chosen by the user's own credential ordering
rather than by my priorities. Neither was observed.
* **The two-screen login regression for ordinary users.** `auth-username-form` +
`auth-password-form` replacing `auth-username-password-form` is a visible change for every user
including non-passkey family accounts, and nobody has logged in through it.
* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
regression check and it is untested. The previous breakage of this exact step
(`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
**Whether the realm import reconciles into an existing realm: measured as NO for the completed
one-shot, but I did not test a re-import.** I did not delete and recreate the
`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
**Treat the re-import path in the README as untested advice, not a runbook.**
**Nothing was proved about the live realm's current state.** I did not read the realm's existing
required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
required-action set actually contains `webauthn-register-passwordless` today; I can only show that
Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
actions to suppress them. **That inference has not been checked against the realm.**
**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
against this change. I reasoned about its assertion by reading it.
**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
needing non-author review.
**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
the gate numbers and the rebase note above were then written into this entry, so the pushed tree
differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
prose edit; CI will gate the pushed commit.
**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
as needing re-measurement.
+108
View File
@@ -0,0 +1,108 @@
---
date: 2026-10-18
branch: fix/coredns-ha
title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
---
## What changed
**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
finding its primary, every operator addressing the API by name. It was invisible because `1/1
Ready` looks perfectly healthy.
It also became urgent rather than latent: the scheduled reboot automation on
`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
single point of failure into a recurring one.
New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
alongside the split-horizon ConfigMap already there.
**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
on the next k3s restart or upgrade. Two alternatives were weighed and rejected:
* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
by hand forever. Too high a price for a replica count.
* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
would both own one object and fight continuously — the exact shape of the `default-deny`
NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).
An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
sees it below `minReplicas` and scales back within one reconcile.
It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
after a k3s restart.
**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
rather than after merge. That hand scale is what this PR makes durable.
## What I verified
* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
* **k3s ownership:** the `AddOn/coredns` object exists with source
`/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
`objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
`platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
Deployment or its replica count.
* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
re-applies on manifest change and on restart.
* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
sit at `minReplicas`.
* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
there.
* All three touched/added YAML files parse.
## What I did NOT verify
* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
`ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
which looks identical to working because the hand scale already left 2 replicas in place. **Check
`kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
**not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
(though harmless).
* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
at 1 replica. Neither the window's length nor the outcome has been measured.
* **I nearly shipped a destructive change.** My first write of
`platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
already existed and I assumed it did not. **No conformance check would obviously have caught the
removal** — it is valid YAML and a legitimate-looking prune.
* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
replicas could in principle land on one node and a single drain would still take DNS down. I did
not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
neither is in this change.
* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
noticing.
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
subset only.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -0,0 +1,76 @@
---
date: 2026-10-18
branch: fix/ha-check-catastrophic-regex
title: "One regex made the merge gate take 28 minutes instead of 2"
---
## What changed
The merge gate started taking **~28 minutes** instead of its usual **<2 minutes**, on every change
touching `infra/**`. One regex was responsible for essentially all of it.
`conformance/tests/l0-static/ha-control-plane-as-code/check.sh` contained:
```python
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
K3S_TEXT, re.S)
```
With `re.S`, every `.` also matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each
consume the entire document. That is **nested unbounded repetition** — ten levels of a quantifier
that can each swallow everything — and a failed or late match backtracks exponentially.
Replaced `.` with `[^\n]*` and dropped `re.S`, so the `{0,10}` is a genuine ten-line window with
nothing to backtrack into. Same intent, same verdict, no DOTALL.
**Why it surfaced now, and it was mine.** `K3S_TEXT` is every `*.yml` in `roles/k3s_server`
concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role
(the kubelet graceful-shutdown config), which grew the input past the point where exponential
backtracking stopped being survivable.** Nothing about the check changed — my change grew its input.
**Why only `infra/**` changes were slow.** `ci/touched-requirements.sh:268` maps `infra/*` to
`R-ENV-01`, and the resulting L0 run is `--include-scope whole-repo --requirements R-ENV-01`, which
pulls in this whole-repo check. Doc-only and `platform/**` changes never selected it, which is why
#69 (1m55s) and #72 (1m53s) stayed fast while #71 (27m44s), the #73 merge (27m46s) and #75 all ran
~28 minutes.
## What I verified
* **The hot spot, by measurement rather than reading.** `docker top` on the live CI job container for
run #299 showed `python3 -` with **19:54 of CPU time** under
`ha-control-plane-as-code/check.sh`, inside
`conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`.
* **Reproduced locally**: the check did not finish within a 100-second budget on an unmodified
checkout.
* **Localised to the exact line** by extracting the inline Python and running it under
`faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` called from
the `join_when` line — not inferred from reading the patterns.
* **The pattern is unique in the repo**: `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh`
returns only this file, so there is no sibling with the same hazard.
* **The fix is fast AND verdict-preserving**, tested against the real `K3S_TEXT` (22,179 chars, 403
lines): the new regex matches (`YES`) in **0.2 ms**. Sanity-checked that the thing it looks for
genuinely exists — one `--server https://` and three `k3s_server_mode == 'secondary'` occurrences.
* **The whole check now runs in 0.199s, rc 0, 115 assertions**, with the same success message as
before.
## What I did NOT verify
* **I did not re-time the full gate in CI after the fix.** The expectation is that `infra/**` changes
return to the ~2-minute range, but that is arithmetic from the 19m54s figure, not a measured
post-fix run. The PR's own CI run is the first real datapoint — read it.
* **I did not audit the other 23 L0 checks for different performance pathologies.** This one was
found because it was actively burning CPU in a live job; a check that is merely slow (not
exponential) would not have stood out, and `ci/gate.sh` has **no per-stage timing** at all
(`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining
time goes.
* **Nothing prevents this class of bug returning.** No check asserts gate stage runtime, and no lint
forbids nested quantifiers under `re.S`. A timing budget per stage, or a `re` lint, would close it;
neither is written here.
* **I did not verify my claim about *why* only `infra/**` was affected beyond the path mapping and
the observed durations.** The correlation is strong (5 runs, two clear clusters) and the live
process list confirms the `--requirements R-ENV-01` scope, but I did not read
`conformance/run.sh`'s scope-selection logic end to end.
* The regex rewrite assumes the `when:` clause is indented with spaces or tabs only (`[ \t]*`). A
file using some other whitespace before `when:` would now fail to match where the old DOTALL
version would have succeeded. Unlikely in YAML, and the current input matches, but it is a
narrowing.
@@ -0,0 +1,334 @@
---
date: 2026-10-18
branch: feat/keycloak-passkey-reconciler
title: "Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated"
---
## What changed
**Why this is a new branch and not an update to PR #72.** The brief for this work said to extend
PR #72 (`feat/keycloak-passkeys`) in place and not open a new PR. That premise was already stale
when I started: **PR #72 merged at 2026-10-18T05:44:04Z** (Gitea API: `state: closed`,
`merged: true`, `merge_commit_sha: 4d489f2`), and `git log origin/main` carries
`4d489f2 Merge pull request '…' (#72)`. A merged PR cannot be updated, and CLAUDE.md §1 leaves
exactly one path to `main`. So this is a fresh branch from `origin/main` in its own worktree, and a
new PR. For the same reason this is a **new** `decisions/` entry rather than an edit to
`decisions/2026-10-17-keycloak-passkeys.md`, which is now merged history describing a different unit
of work; it is the entry to read first for the passkey *design* decisions, which are not repeated
here.
**The problem.** PR #72 declared the WebAuthn passwordless policy and a restructured `browser-mfa`
flow in `platform/keycloak/keycloak-realm-import.yaml`. That declaration **could not take effect**.
A `KeycloakRealmImport` is a one-shot; prod's reads `Done=True / Started=False / HasErrors=False`,
so the operator will never re-run it and every realm-level field or flow declared after the first
import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it
changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the
2026-09-06 client-secret incident, third field over.
**What now makes it converge.** A **sibling** reconciler, modelled on
`platform/keycloak/client-secret-reconciler.yaml`:
- `platform/keycloak/passkey-reconciler.yaml` — ConfigMap (Python, stdlib only, **zero dollar
signs**) + CronJob on `*/15 * * * *`, `automountServiceAccountToken: false`, admin credential by
`secretKeyRef` only. Three passes, **all of which converge**:
1. **WebAuthn passwordless policy.** Strict read-modify-write of `GET`/`PUT /admin/realms/homelab`,
changing only the `webAuthnPolicyPasswordless*` keys that differ, readback-verified.
2. **`webauthn-register-passwordless` required action.** Strict read-modify-write of one
**literally-named** alias; converges `enabled: true` / `defaultAction: false` only.
3. **The browser authentication flow — BLUE/GREEN.** Build the declared tree under a new, unused
alias; verify it by readback; flip `browserFlow` in **one atomic PUT**.
- `ci/render-keycloak-passkey-policy.py` + the **generated**
`platform/keycloak/passkey-desired-state.yaml` — the desired state, rendered from the realm
import, `--check`ed by the merge gate.
- `conformance/tests/l0-static/keycloak-passkey-reconciler/` — `check.sh`, `chainsaw-test.yaml`
(`scope: whole-repo`), `mutations.py`.
- `platform-config/kustomization.yaml` — both new resources, in the **same** Kustomization as the
realm import.
- `platform/keycloak/README.md` — the layout table, the section heading, and the "does NOT reach the
live realm" section rewritten: all three declared things now converge, with the blue/green
mechanics, the rollback command and the `--no-bind` fallback written out.
### The flow design was reconsidered, and the first version of it was wrong
**This is worth recording because the superseded reasoning was not silly, it was just wrongly
scoped.** The first design of pass 3 was **audit-only**: compare the live browser flow against git,
report divergence loudly, and never write it — leaving the restructure as a one-time admin-console
action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow,
create-execution, set-requirement and bind-config as *separate* untransacted admin-API calls, so a
run dying between them leaves a **half-built flow bound as the login path** for every SSO-gated app
*and* kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the
conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every
platform-admin.
**That risk analysis was sound but it assumed the wrong shape.** It is true only if you mutate the
*bound* flow. The operator's instruction was *"if this can be done through GitOps, it should"*, and
audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already
forces. So:
- All the multi-call construction happens on a flow **nothing is using** (the declared alias plus a
hash of the desired shape, applied to *every* alias in the tree — reusing `browser-mfa-forms`
would mean writing *into* the bound tree). A failure at any point leaves an orphaned, unbound flow
and **zero user-visible effect**. The untransacted-calls problem dissolves.
- The built flow is **read back and verified** — nesting levels, ordering, authenticators,
requirements, and the conditional-OTP config *contents* — **before** anything is bound. The
regression the operator feared is now caught *before* it can bite rather than after.
- The flip is **one atomic PUT** of the realm setting `browserFlow`.
- **Rollback is one write.** The previous alias is printed in the run log, and there is **no
`DELETE` anywhere in the allow-list**, so the old flow survives as a working rollback target.
**The new safety property, enforced structurally rather than by intention.** Blue/green needs
flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone
cannot express "but not the flow that is bound", because its write entries are alias-parameterised.
So a second guard sits at the same choke point in `_http()`: the live `browserFlow` alias and its
**entire subtree** of sub-flow aliases and execution ids are read first, and every write is screened
against those sets before a socket opens. Protecting the whole subtree matters — adding an execution
to `browser-mfa-forms` changes live login behaviour just as surely as touching `browser-mfa`. It is
**fail-closed**: the sets start each run as "not determined", and while they are, *every* flow write
is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass
writes nothing.
**Idempotence became load-bearing rather than merely nice.** A rebuild-and-reflip every 15 minutes
would churn the login path on a schedule, so the bound flow is compared *structurally* and a match
produces **zero writes and no new flow**. Sub-flow and config *aliases* are deliberately not
compared: a flow this program built carries a hash suffix, so comparing labels would make it
permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is **reused**; a
half-built one is left alone and the next candidate alias is used, with the candidate count
**bounded** so a persistently failing build fails loudly instead of creating unbounded sprawl.
**Where the rollback target is recorded, and why not as an annotation.** The brief suggested
recording the previous `browserFlow` alias as an annotation on the CronJob or in a ConfigMap. The
pod **cannot**: it runs with `automountServiceAccountToken: false` and has no Kubernetes API access
at all, which is the same property that made the mounted-generated-ConfigMap design preferable to
reading the CR (and is why this change adds no `**/rbac.yaml`). Giving it write access to its own
annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse
bargain than the one it would buy.
So the record is in two places that cost nothing: the **run log** prints
`ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted.`, and — more
durably — **the realm itself**, because nothing is ever deleted. Every superseded flow is still
listed under `Authentication -> Flows`, so the rollback target is discoverable from the admin console
even if every Job log has rotated away. The README carries the rollback procedure.
**I achieved full convergence including the bind.** `--no-bind` exists as the deliberate fallback
(build and verify, leave the flip as a one-write human act) but it is not the default and was not
needed.
**No `**/rbac.yaml` was touched.** The brief floated having the reconciler read the
`KeycloakRealmImport` CR from the Kubernetes API, which would need a Role, a RoleBinding and a
ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state
**generated** from the import and **mounted**, so the pod keeps `automountServiceAccountToken:
false` and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with
`--check`, not by a second declaration being trusted.
**Sensitive paths touched** — checked against `infra/gitea/approval-policy.json` rather than from
memory, which caught one I had initially missed. **Three**:
| Path | Why it is in the policy | What I changed |
|---|---|---|
| `ci/**` | the pipeline itself | added `ci/render-keycloak-passkey-policy.py` |
| `conformance/**` | changing the tests changes what green *means* | added the L0 test directory |
| `platform-config/kustomization.yaml` | Flux wiring / which manifests are actually applied | added the two new resources |
**Not** touched: `**/rbac.yaml`, `clusters/**`, `platform/kyverno/**`, `.gitea/**`, `bootstrap/**`,
`infra/gitea/**`, `.sops.yaml`, any `*.sops.yaml`, `CLAUDE.md`.
So this PR **requires an approving review from someone who is not its author**, and
`ci/approval-policy.sh` is fail-closed, so it will block until that exists. I did **not** post
`/approve-override` — CLAUDE.md §1 names doing that on your own agent PR as exactly the
self-approval the stage exists to prevent. The operator needs to either review it or post the
override.
## What I verified
**Measured against the live cluster** (read-only `kubectl --context k3s-prod`):
- `keycloakrealmimport homelab`: `Done=True Started=False HasErrors=False` — the import is a
completed one-shot. **This is the premise of the whole change and it is measured, not assumed.**
- Keycloak image actually running: `quay.io/keycloak/keycloak:26.0.6`.
- `kustomization/platform-config` `.status.lastAppliedRevision` ==
`gitrepository/flux-system` artifact revision == `main@sha1:83d3568…`, which contains the PR #72
merge. So the inert declaration is genuinely applied and genuinely inert.
**`PUT /admin/realms/{realm}` does not touch users, groups, clients or roles in 26.0.6** —
established from source at tag `26.0.6`, not from documentation or memory:
- `RealmAdminResource.updateRealm` is the only handler. Its javadoc: *"Any user, roles or client
information in the representation will be ignored. This will only update top-level attributes of
the realm."*
- It delegates to `RepresentationToModel.updateRealm` →
`DefaultExportImportManager.updateRealm` (`model/storage-private/.../datastore/`), **lines
725–894**. I brace-matched that body and enumerated every `rep.getX()` call in it: it contains
**no** `getUsers()`, `getClients()`, `getGroups()`, `getRoles()`, `getRequiredActions()`,
`getAuthenticationFlows()` or `getAuthenticatorConfig()`. Those objects are **unreachable** from
the endpoint, not merely ignored by convention.
- Two findings that changed the design, both from the same read:
- `updateRealm` **does** read `rep.getBrowserFlow()`, so the endpoint can rebind the realm's login
path. The write therefore carries `browserFlow` straight back from the GET and a hard assertion
refuses the PUT if it would differ.
- The passwordless policy is applied **all-or-nothing and only when `RpEntityName` is non-empty**
(line 842), and `getWebAuthnPolicyPasswordless(rep)` substitutes a **Keycloak default** for every
null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and
one missing `RpEntityName` would drop the whole policy while returning 204. This is why the write
is a strict read-modify-write of the full live representation.
- `AuthenticationManagementResource.updateRequiredAction` (line 1205) is **per-alias** and likewise
**replaces** the provider from the representation (name, alias, priority, enabled, defaultAction,
config), and 404s when the alias is unregistered. Hence read-modify-write there too.
- `DefaultRequiredActions` registers `webauthn-register-passwordless` with `enabled=true`,
`defaultAction=false`, `priority=80` — confirming the realm import's claim that declaring it would
restate a default while deleting ten siblings.
- **The realm GET really does return what the read-modify-write needs.** This is load-bearing: if
the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset
them to defaults. `ModelToRepresentation.toRepresentation` (lines 471–482) sets **all ten**
`webAuthnPolicyPasswordless*` fields plus `ExtraOrigins`, and line 496 sets `browserFlow`. I
checked for an `internal`/`export` guard around that block — there is none in lines 380–500, so
those fields are emitted unconditionally, not only on the export path.
**The five flow-construction endpoints, read from 26.0.6 source before being used:**
- `POST /authentication/flows` — `AuthenticationManagementResource.createFlow`, rejects a duplicate
alias. Carries **no alias in its path** (the alias is in the body), which is why it needed its own
body-based screen; without that it escaped the protected-subtree guard entirely. My selftest
caught that, not my reading.
- `POST /flows/{alias}/executions/flow` — `addExecutionFlow`, body `{alias, type, description}`.
Creates the sub-flow **at requirement DISABLED**, and refuses if the parent `isBuiltIn()`.
- `POST /flows/{alias}/executions/execution` — `addExecutionToFlow`, body `{provider}`. Also
DISABLED unless the factory has exactly one requirement choice. Both assign `getNextPriority()`,
so **insertion order is the resulting order** — which is why the builder inserts depth-first in
declared order and then verifies by readback.
- `PUT /flows/{alias}/executions` — `updateExecutions`. **A trap I would have walked into:** it
compares `rep.getPriority()` against the stored priority, and a JSON body omitting `priority`
deserialises it as the int **0** — which would silently **reshuffle the whole execution order**
while only a requirement change was intended. Every requirement write therefore carries the
execution's current priority back. There is a mutation for it and a selftest assertion on the
resulting order.
- `POST /executions/{id}/config` — `newExecutionConfig`, creates an authenticator config and binds
it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and
the old flow already uses the declared alias.
**Offline proofs, all green:**
- The reconciler's `--selftest`, **extracted from the shipped ConfigMap** and run: **216 assertions
pass**. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the
all-or-nothing policy rebuild, the required-action replace, the `updateExecutions` priority trap,
and flow construction where everything is created DISABLED — so these are tested against the real
semantics rather than a forgiving fake.
- **A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds
and never touches the live flow.** Each case asserts the realm is still bound to the old alias,
that no write targeted the bound subtree, and that every live flow survives. This is the assertion
that makes blue/green a property rather than a claim.
- **Verify-before-bind is exercised, not assumed.** Two stub sabotages — a server that accepts the
requirement writes and applies the wrong value, and one that writes the conditional-OTP role under
`condition.evaluated.user.role` — both end with the flow **built but NOT bound** and the old flow
still live.
- `mutations.py`: **all 30 injected regressions caught**, including the six that are the blue/green
safety argument itself (build into the bound flow; never call the screen; reset the protected sets
to empty; bind before verify; report a failed build as success; reuse an existing flow without
verifying it).
- Mutation testing again found real gaps in my own work, which I fixed rather than weakening the
mutations. Across the two designs: no readback coverage on the realm write, no coverage of an
undeclared policy key surviving, a mis-indented anchor, and — in this design — **no coverage at
all of the bound flow being unreadable**, which is the fail-closed path. It also showed me that
**two of my proposed mutations were not real regressions**: the module-level initial value of the
protected sets is dead code (`run()` resets them first), and removing the early return after a
build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that
*are* load-bearing rather than pretending the originals had been caught.
- The new L0 `check.sh`: rc 0, all assertions. It asserts the renderer `--check`; that the realm
import and the desired state are rendered by the **same** Flux Kustomization; that the renderer's
and reconciler's writable key tuples are identical (parsed out of both sources with `ast`); that
`defaultAction` is `false`; that the realm import declares **no** `requiredActions` block
(previously only a comment saying "DO NOT ADD ONE"); that the **sibling** reconciler's allow-list
was not widened; that **nothing is deletable** and the execution-/config-id-scoped write endpoints
stay off the list (they carry no flow alias, so they could not be screened); and — the
load-bearing one — that the **protected-subtree guard exists in the source, is wired into
`_http()`, resets to "undetermined" every run, screens the body-only create-flow endpoint, and
that the policy write still refuses to rebind `browserFlow`**.
- `ci/render-keycloak-passkey-policy.py --write` output round-trips: the generated JSON is accepted
by the reconciler's `declared()`, its 10 policy keys equal the reconciler's `POLICY_FIELDS`, and
it flattens to the exact 11-step tree the realm import declares, with
`conditional-otp-platform-admin` bound at level 4.
- `${BASE_DOMAIN}` is carried into the generated ConfigMap **verbatim**, and both files sit in
`platform-config`, so one substitution pass decides the RP ID.
- Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed
manifest.
**The conditional-OTP regression is detectable — and now also preventable.** The comparison is
key-by-key on the authenticator config contents, and because it runs **before** the bind, a flow
carrying the bug never becomes the live login path. `--selftest` covers it three ways: the historical
`condition.evaluated.user.role` key injected into a *built* flow (built, verified, **not bound**,
old flow still live); the same bug already live in the *bound* flow (reported, then **fixed** by
building and binding a correct flow); and the config disappearing entirely. `mutations.py` deletes
the comparison and the undeclared-key report and requires the selftest to notice both. The suite
also covers a requirement flipped from `ALTERNATIVE` to `REQUIRED`, the un-restructured legacy flow,
a correct-but-unbound flow, and a cyclic declaration.
**Gate:** see the gate section of the PR / the run quoted in my report — rc and per-stage results
are recorded there rather than paraphrased here.
## What I did NOT verify
- **The live realm's actual passkey state is entirely UNVERIFIED.** I never authenticated to the
Keycloak admin API. Doing so needs the `keycloak-admin-credentials` password, which CLAUDE.md §6
forbids decrypting or printing, so I did not read the realm's current
`webAuthnPolicyPasswordless*` values, its required-action list, or its `browser-mfa` executions. I
therefore **do not know** how much drift the first real run will report, nor that the live flow is
in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift
the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
- **Nothing has ever run against a real Keycloak.** Every behavioural proof is against an
in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This
matters more for the blue/green flow builder than for anything else in the change, because it is
**five endpoints I have never exercised**. Specifically unproven in reality:
- That `POST /flows/{alias}/executions/flow` and `.../executions/execution` accept the bodies I
send and assign priorities in insertion order. If they do not, the build produces a tree that
fails verification — which is the safe failure (nothing is bound, login unaffected), but it
means **the flow would never converge** and the Job would fail every 15 minutes.
- That `PUT /admin/realms/homelab` accepts a full round-tripped representation. The analogous
*client* endpoint rejects one with `400 "Cannot parse the JSON"` (measured 2026-09-06), and the
sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm
endpoint differs; no measurement here confirms it. **If it does not, both the policy pass and
the atomic bind fail closed** — loudly, writing nothing.
- The exact JSON shapes compared: `AcceptableAaguids` as `[]` rather than absent or null,
`CreateTimeout` as a JSON int, `RequireResidentKey` as the literal string `"Yes"`. A shape I did
not anticipate would show up as **perpetual drift** — a write every 15 minutes that never
converges — rather than as an error.
- That `authenticationConfig` on an execution-info response is the config **id** (I read it from
the representation class, but never saw a live response).
- **I did not verify the alias-collision behaviour against a real realm.** The recovery logic
assumes `POST /flows` returns a non-2xx on a duplicate alias (the source checks
`getFlowByAlias != null` and raises `ErrorResponse.exists`), and that a config alias collision
likewise fails. If either silently succeeded, the recovery path would behave differently than
modelled.
- **Flow sprawl is bounded but real.** Every genuinely new desired shape creates a new flow and
**nothing is ever deleted** — by design, because the old flow is the rollback target. Over many
shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented
beyond "an operator deletes them". I consider that the right trade (deletion is the one operation
that could destroy a rollback target) but it is a cost, not a non-issue.
- **No L2 behavioural test was added.** The reconciler has an `--audit` mode built for one and the
sibling has `tests/l2-behavioral/keycloak-client-secrets-match-git`, but I added no equivalent.
And `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 would not run in CI anyway. **CI proves the
offline subset only; it proves nothing about the live realm.**
- **No alerting.** Unlike the first design there is no longer any failure that is "expected by
design", so a PrometheusRule on `kube_job_status_failed` would now be meaningful — and I did not
add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret
key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
- **That passkey login works.** No passkey has been enrolled and no browser has been pointed at
this. Nothing offline can prove a WebAuthn ceremony completes, that the `ALTERNATIVE` fork
degrades to the password form via `CREDENTIAL_SETUP_REQUIRED`, or that a platform-admin is still
asked for TOTP on the password branch. The README's four-step manual check remains the only thing
that would establish it, and it remains undone.
- **That the RP ID is right for the origin.** `${BASE_DOMAIN}` is a legal registrable-domain suffix
of `keycloak.${BASE_DOMAIN}` on paper, but a wrong RP ID fails **in the browser**, before Keycloak
is involved, with no server-side log line. The reconciler will happily converge the realm on a
wrong RP ID; it checks that git and the realm agree, never that git is correct.
- **The first real run will change how everyone logs in.** Once this merges and Flux applies it, the
next tick builds a flow and — if it verifies — **flips the realm's browser flow**. The UX change
is deliberate and documented (username screen, then password), but it is the first time it will
have happened, unattended, on a schedule. The rollback is one write and the old flow survives;
that is the mitigation, and it has not been rehearsed either.
- **CI's own result.** My gate run is local. `gitea-settings` cannot run locally, and
`ci/approval-policy.sh` reports green N/A offline while this branch touches three sensitive paths,
so **the PR is guaranteed to need a non-author approving review** and I have not seen CI report. I
did not and will not post `/approve-override` on my own PR.
- I re-measured branch protection this session (`protected: true`, `user_can_push: false`,
`enable_status_check: true`, context `gate / gate (pull_request)`, `required_approvals: 0`), but
the branches endpoint does **not** expose `block_admin_merge_override`, so that row of CLAUDE.md
§2 is **not re-confirmed** by me.
+82
View File
@@ -0,0 +1,82 @@
---
date: 2026-10-18
branch: fix/pihole-local-ttl
title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
---
## What changed
The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
**asymmetry in cacheability** that we created.
Measured on prod 2026-10-18:
```
dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
```
dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
rather than forwarding live. So:
* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
lookup, which is maximum exposure to any momentary gap;
* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
(standard forwarder behaviour when the forward target does not answer) it latches the public WAN
address and serves it to **every LAN client** for five minutes.
So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
regenerated with the canonical render command.
## What I verified
* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
143 (decrementing), which a pure forwarder could not do.
* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
address), so a fallback always has a wrong answer available.
* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
`/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
`FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
--no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
`apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
## What I did NOT verify
* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
nor the router caching the private answer, nor the flap stopping. **Verify after merge with
`dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
— so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
public answer latches again. That is a real availability problem, not a caching one.
* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
(`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
(tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
that decision, and it is deliberately **not** in this change.
* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
there. Only this one workstation's resolver chain was inspected.
* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
`home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
next recurrence would again be found by a human noticing. That gap is not closed here.
@@ -0,0 +1,78 @@
---
date: 2026-10-18
branch: fix/renovate-cron-window
title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
---
## What changed
`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
never have produced a single PR.**
Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
re-checks it and declines to open PRs outside it. So:
```
cron 03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local
inside? -> NO (3h45m before the window opens)
```
The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
green tick forever, indistinguishable from "nothing needed updating". That is the same
"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
pending updates PR #73 was written to surface.
Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
position moves:
| | local window | in UTC |
|---|---|---|
| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
A cron cannot follow DST, so this is the only way to get it right once.
## What I verified
* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
`America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
both inside `03:00–06:00`. Computed, not reasoned.
* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
`timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
when PR #73 was authored.
* The edited workflow parses as YAML.
## What I did NOT verify
* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
usable PR.
* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
`customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
change it**, because guessing at a config key that could fail validation and take the whole run
down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
validation warning naming `regex`.
* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
schedule". That makes the first dispatch's output less informative than it looks; to see what
Renovate *would* do, either dispatch inside the local window or override the schedule for that
run. Not tested either way.
* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
does.
* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
return. A static check comparing the two is feasible and is not written.
@@ -0,0 +1,78 @@
---
date: 2026-10-18
branch: fix/renovate-invalid-config
title: "renovate.json was invalid, so the first manual run errored out"
---
## What changed
The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
**`renovate.json` was never a valid Renovate config.**
It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
top-level options**, so the entire config failed validation. Proven with the official validator
rather than inferred:
```
$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
ERROR: Found errors in configuration
Invalid configuration option: _comment
Invalid configuration option: _docker_comment
Invalid configuration option: _ignorePaths_comment
Invalid configuration option: _kubernetes_comment
Invalid configuration option: _unmanageable_comment
```
This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
happens"; #73 shipped both.
**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
is preserved as comments positioned next to what it explains, which a flattened `description`
string would have lost.
A header on the new file records the failure, the validator command, and the instruction to run it
before committing — this class of bug is invisible to the repo's own gate.
## What I verified
* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
from the workflow.
* **The fix validates**: the same validator against `renovate.json5` returns
`INFO: Config validated successfully against 1 file(s)`.
* **Nothing was dropped in the conversion**, checked programmatically against `git show
HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
prints a normalisation diff that mentions it, which is its own migration suggestion, not a
removal.
* **74 comment lines survive** in the new file, so the rationale is intact.
* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
the workflow's fail-hard preflight is satisfied.
* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
`custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
validator accepts `regex`, so that concern was unfounded and nothing here changes it.
* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
conformance/` returns only comment prose.
## What I did NOT verify
* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
bot works — only that these two specific defects are fixed.
* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
`dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
`prHourlyLimit: 2` / `prConcurrentLimit: 5`.
* **Whether the managers actually extract anything.** A valid config that matches no files is still
a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
`managerFilePatterns` are right, and I have not seen them.
* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
discovery is long-standing and the validator accepted the file, but the workflow's image tag was
not cross-checked against the version that added it.
* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
here, and `conformance/**` is a sensitive path.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -0,0 +1,172 @@
---
date: 2026-10-29
branch: feat/keycloak-passkey-reconcile
title: "Passkey reconciler review fixes: bind only on git's say-so (rollback sticks), protect the real 26.0.6 sub-flows, screen the requirement PUT by body id, alert on failure"
---
## What changed
This branch carries the unmerged passkey reconciler (commit `36fcb7b`, cherry-picked with its
authorship, recorded in `decisions/2026-10-18-keycloak-passkey-reconciler.md`) plus the fixes for a
review of it. Read that entry for the design; this one records what the review found and what
changed.
**1. HIGH: the bind was automatic, and rollback did not stick.** Within one tick of merge the
reconciler would build and BIND `browser-mfa-<hash>`, a login flow no human had tried. Worse, the
README's rollback (rebind `browser-mfa`) was undone every 15 minutes: the next run found the
verified candidate again and rebound it, and a hotfix inside the bound flow was "corrected" by
building and binding an `-r2`.
- New git-declared switch: `bind-browser-flow` in `platform/keycloak/passkey-desired-state.yaml`,
a second key of the same ConfigMap, so it reaches the pod through the existing mount. **Default
`"false"`.** It is the one hand-edited line in a generated file: the renderer carries it through
on `--write`, and `--check` (run by the L0 gate) refuses anything but the quoted strings `"true"`
/ `"false"`. An unquoted `true` is a YAML boolean, which a ConfigMap rejects at apply time and
which would have failed the whole `platform-config` Kustomization, realm import included.
- `"false"`: pass 3 builds and verifies the candidate (or reuses a verified one), logs
`WOULD BIND <alias>`, exits 0 with `KEYCLOAK_PASSKEY_RECONCILED_OK_PENDING_BIND`, and never PUTs
a changed `browserFlow`. That holds after an earlier bind too, so the operator's rebind sticks.
If the bound flow *is* the candidate alias and has been hand-edited, it is left alone (no `-r2`
is built beside it).
- `"true"`: binds as before, and keeps asserting it. That is GitOps working; the README says so,
and says not to hotfix the flow while it is `"true"`.
- The CronJob passes no flag. `--no-bind` still exists for a hand-run Job and can only narrow
git's value. The L0 check asserts the CronJob carries no `--` argument.
- Rollback is now: a PR setting `"false"` (first), then one rebind write. README documents the
order and why it matters, plus how to test the candidate on one client (a per-client
`account-console` browser-flow override) before the go-live PR.
**2. HIGH: the protected-subtree guard read the wrong field for sub-flows.** Read from source at tag
26.0.6 (`services/src/main/java/org/keycloak/services/resources/admin/AuthenticationManagementResource.java`,
fetched from GitHub this session): `recurseExecutions` sets `displayName = flowRef.getAlias()` and
`flowId` on a sub-flow row and never calls `setAlias`. `alias` is set only on authenticator rows,
where it holds the bound **config's** alias. So the protected set was `{bound}` alone, and a write
into `browser-mfa-forms` would not have been refused. The old stub returned `alias` on sub-flow
rows, which is why all 216 assertions passed anyway.
- New `subflow_alias()` reads `displayName` and cites the source. I did not resolve by `flowId`:
`getFlows()` (same file) filters to `isTopLevel()`, so a nested sub-flow's id resolves to nothing
through the flows list. A sub-flow row with no `displayName` makes the listing unreadable, which
leaves the flow pass fail-closed.
- The stub now returns rows shaped like 26.0.6's: sub-flow rows carry `displayName`, `flowId` and no
`alias`/`providerId`. Authenticator rows carry a display *type* as `displayName` (not the
provider id) and the config alias as `alias`. Nulls are omitted.
- New assertions: after a real run, `browser-mfa-forms` and `browser-mfa-conditional-otp` are in
the protected set, and an execution add or requirement write into them is refused. A missing
`displayName` fails closed.
**3. MEDIUM: `PUT /flows/{alias}/executions` was screened on the wrong field.** `updateExecutions`
resolves `realm.getAuthenticationExecutionById(rep.getId())` without checking membership in
`{alias}`. The same handler also renames the flow named by `rep.getFlowId()` when the body carries
`authenticationFlow` + `displayName`. Now:
- `body["id"]` is screened against the protected execution ids.
- It must also be in `_BUILD_EXECUTION_IDS`: the ids read back from the flow this call just
created, open only inside `build_flow`.
- The config `POST /executions/{id}/config` is held to the same build scope.
- The body may carry only `id`, `requirement` and `priority`.
- Selftest cases cover each of these.
**4. Stale docs on sensitive paths** now describe what ships, including the switch:
`platform-config/kustomization.yaml`, the chainsaw test header, the desired-state header and its
renderer template, the renderer's own header and `--check` message, and the README.
**5.**
- `backoffLimit: 0`: every failure is deterministic within a tick, and a retried build burns
another of the 5 candidate aliases.
- New alert `KeycloakPasskeyReconcilerFailing` in `platform/monitoring/alerts.yaml` (warning,
`for: 15m`, `kube_job_status_failed` on `job_name=~"keycloak-passkey-reconciler-.*"`), the same
shape as the sibling's. It has promtool tests in
`platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`, in both directions plus
"the sibling's failure is not this alert" and "one transient failure stays quiet". The alert
ratchet in `alert-rules-unit-tested` would have refused it untested. A new runbook,
`runbooks/keycloak-passkey-reconciler.md`, is projected by the runbooks generator.
- The stale "Renovate does not manage plain `image:` tags" comment is corrected: `renovate.json5`
enables the `kubernetes` manager over `platform/**`. The image is pinned to
`python:3.13.15-alpine3.23@sha256:6438599575cca0d1df94aeee0d2ae088d4d8846eab554b2ee7784a3a6df0d516`,
the multi-arch index digest read from Docker Hub today. The tag is mutable: the sibling
reconciler's pod on prod reports `imageID ...@sha256:75f27d68...` under the same tag, which is
neither today's index nor today's amd64 manifest.
**6. Hygiene.** The ~160-line header that appeared in both the YAML and the script is now kept once,
in the script, where it ships next to the code and its selftest. A short header above the
ConfigMap points to it. Nothing else was restructured, apart from the build's second half moving
into `_finish_build()` so the build scope can be opened and closed around it.
Sensitive paths touched: `ci/**` (renderer), `conformance/**` (L0 check, mutations, chainsaw),
`platform-config/kustomization.yaml` (comment only). This needs the operator's
`/approve-override` or a non-author review. I did not post one.
## What I verified
- **Local toolbox gate, rc 3** (pinned toolbox via `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE
shim bind-mounting the main `.git`, run on the committed branch). Stages: `spec-immutability`
pass (the spec repo resolved as a sibling), `decisions-log` pass, `lint` pass, `l0` pass
(including `keycloak-passkey-reconciler`, `alert-rules-unit-tested` and
`alert-runbooks-resolvable`), `render-golden` pass, `l1` pass, `approval-policy` pass (N/A
offline; it is guaranteed to fail in CI). **`gitea-settings` did not run** (no
`SETTINGS_API_TOKEN` locally) and is the only stage that did not run. L2/L3 `conformance` is
skipped by design (no cluster). rc 3 is **not** green.
- `platform/monitoring/tests/run.sh platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`
in the toolbox: `promtool check rules` SUCCESS (54 rules), `promtool test rules` SUCCESS,
promtool 3.14.0.
- **Selftest: 259 assertions pass** (was 216), extracted from the shipped ConfigMap by the L0
check.
- **`mutations.py`: 40 of 40 injected regressions caught** (was 30). The ten new ones each undo
one fix:
- sub-flows read by `alias` again;
- an unnameable sub-flow tolerated;
- the requirement PUT screened on its path only;
- execution writes not held to the build tree;
- extra body keys allowed;
- the bind switch ignored in `run()`;
- the bind switch ignored in `reconcile_flow()`;
- an absent switch reading as true;
- an invalid switch guessed at;
- `-r2` built beside a hand-edited bound flow.
"Sub-flows by `alias`" is caught less cleanly (the selftest aborts on an unreadable listing
rather than failing an assertion). It is still caught.
- The renderer accepts `"true"` and `"false"` and refuses `true` (unquoted) and `"yes"`. I
checked this by hand-editing the committed file and running `--check`.
- The 26.0.6 source for `recurseExecutions`, `updateExecutions` and `getFlows` was read directly
(lines ~225-231, 658-724 and 747-806 of the file at that tag).
- Read-only `kubectl --context k3s-prod`: Keycloak pods run `quay.io/keycloak/keycloak:26.0.6`
(2 replicas). The only CronJob in `keycloak` is `keycloak-client-secret-reconciler`, so the
passkey reconciler is **not** live.
## What I did NOT verify
- **Nothing has run against a real Keycloak.** The stub's row shapes now follow the 26.0.6
source, but they are still my reading of that source, not a captured response. In particular,
I have never seen a live executions listing to confirm `displayName` on a sub-flow row. If it
differs, the failure is fail-closed (no build), not fail-open.
- **The candidate flow has never been tested in a browser.** That is now a precondition of the
go-live PR (README, "Testing the candidate flow before binding it"). Nobody has run those steps.
I also have not checked that the per-client `account-console` override behaves as described on
26.0.6.
- **"Merging changes nothing users see" is true of login, but passes 1 and 2 still write the
realm.** They set the passwordless policy and enable the enrolment action. That action is
enabled by default in 26.0.6, but I did not read the live realm, because no admin API calls were
allowed. Pass 3 also creates the unbound candidate flow (realm DB writes, invisible to login).
- **Race in pass 1, not fixed:** the policy PUT is a read-modify-write that carries `browserFlow`
back from its own GET. An operator rebind landing between that GET and PUT would be reverted.
The window is one HTTP round trip and only exists when the policy has drifted.
- **The alert has never seen a real series.** The CronJob has never run, so
`kube_job_status_failed{job_name=~"keycloak-passkey-reconciler-.*"}` does not exist yet. Its
shape is copied from the sibling's rule, which does have live series.
- **The digest is of today's index.** I did not pull it or run the image. Renovate's handling of a
`tag@digest` pin under the `kubernetes` manager is from its documentation, not from a Renovate
run here.
- **CI's own result** is in the PR, not here. `approval-policy` is expected to fail until the
operator overrides.
### Follow-ups (out of scope here, deliberately not done)
- A dedicated least-privilege service-account client for the reconciler, instead of the
master-realm break-glass admin it shares with the client-secret reconciler.
- A NetworkPolicy for the `keycloak` namespace.
- Whether a passkey login should skip TOTP for platform-admins. This is the PR #72 design
decision, currently "no TOTP after a passkey".
- A `KeycloakPasskeyReconcilerStale` rule (no successful run in N hours), like the sibling's.
Only the failure alert was in scope.
@@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev"
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
# not finish detaching came back `degraded` and had to rebuild.
#
# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
# rather than the only thing standing between a reboot and a hard kill.
#
# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
# well under 90s here, and Longhorn needs only enough time to flush and detach.
k3s_kubelet_shutdown_grace_period: "90s"
k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
# Set a little above the kubelet's total (90s + 20s) to leave headroom.
k3s_logind_inhibit_delay_max_sec: 120
@@ -15,3 +15,12 @@
port: "{{ k3s_api_port }}"
timeout: 180
listen: Restart k3s
# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
- name: Restart logind
ansible.builtin.systemd_service:
name: systemd-logind
state: restarted
listen: Restart logind
@@ -86,6 +86,16 @@
# k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
# BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
# config.yaml is unchanged and the Restart handler does not fire.
# ── Graceful node shutdown ──────────────────────────────────────────────────────────────
# Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
# containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
# 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
# InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
# quietly ignored.
kubelet-arg:
- "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
- "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
kube-apiserver-arg:
- "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
- "oidc-client-id=kubernetes"
@@ -183,3 +193,46 @@
host: "127.0.0.1"
port: "{{ k3s_api_port }}"
timeout: 180
# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
# before proceeding regardless. A grace period above that cap is therefore silently truncated --
# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
# instead of documenting it, because the failure is invisible.
- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
ansible.builtin.assert:
that:
- (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
fail_msg: >-
k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
quiet: true
tags: [shutdown]
- name: Ensure /etc/systemd/logind.conf.d exists
ansible.builtin.file:
path: /etc/systemd/logind.conf.d
state: directory
owner: root
group: root
mode: "0755"
tags: [shutdown]
# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
# would fully revert the change.
- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
ansible.builtin.copy:
dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
owner: root
group: root
mode: "0644"
content: |
# Managed by Ansible (roles/k3s_server). See defaults/main.yml.
# MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
[Login]
InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
notify: Restart logind
tags: [shutdown]
+265
View File
@@ -0,0 +1,265 @@
#!/usr/bin/env bash
# =================================================================================================
# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
# back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
# =================================================================================================
# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
# sequence, so the next person does not improvise it.
#
# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
# shutdown, for three reasons the kubelet's drain does not cover:
# 1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
# 2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
# that Flux will not set or clear for you.
# 3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
# Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
# place avoids all of it.
#
# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
#
# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
set -euo pipefail
CONTEXT="${CONTEXT:-k3s-prod}"
KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
TIMEOUT="${TIMEOUT:-600}"
# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
log() { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
ok() { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
die() { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
# ---------------------------------------------------------------- discovery
# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
# them down would remove the very machinery that performs the detach (longhorn-system), keep the
# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
# kubevirt).
pvc_workloads() {
"${KUBECTL[@]}" get pods -A -o json | python3 -c '
import json,sys
PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
out={}
for p in json.load(sys.stdin)["items"]:
ns=p["metadata"]["namespace"]
if ns in PROTECT: continue
if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
ow=(p["metadata"].get("ownerReferences") or [])
if not ow: continue
k,n=ow[0]["kind"],ow[0]["name"]
if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
elif k=="StatefulSet": k="statefulset"
else: continue # Jobs, and CNPG Clusters, are handled separately
out[(ns,k,n)]=1
for ns,k,n in sorted(out): print(ns,k,n)
'
}
# ---------------------------------------------------------------- the PVC-deletion guard
# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
# first misread as a miscount.
#
# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
# an enormously better outcome than a deletion, and that is the whole trade being made here.
sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
local policy
policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
-o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
[ "$policy" = "Delete" ]
}
wait_for() { # wait_for <description> <shell-condition>
local desc="$1" cond="$2" waited=0
log "waiting: $desc (up to ${TIMEOUT}s)"
until eval "$cond"; do
(( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
sleep 10
done
ok "$desc"
}
# ---------------------------------------------------------------- quiesce
do_quiesce() {
need kubectl; need python3
# Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
# how one replica becomes zero; better to stop here and let it rebuild first.
local unhealthy
unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
-o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
[ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
ok "Longhorn: all volumes healthy"
# 1. Flux first, or it reverts every scale-down below within its reconcile interval.
log "suspending Flux Kustomizations"
local k
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
done
ok "all Kustomizations suspended"
# 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
# pod eviction would be a power cut to the VM.
if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
log "ACPI-stopping the UniFi VM"
"${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
fi
# 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
# at a time, primary last-ish, so this is not instant.
log "hibernating CloudNativePG clusters"
local c ns name
for c in "${CNPG[@]}"; do
ns="${c%%/*}"; name="${c##*/}"
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
done
# 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
# TrueNAS mount; PiHole last because it is the LAN resolver.
log "scaling down PVC-holding workloads"
local line ns2 kind nm deferred=() skipped_delete_policy=()
while read -r ns2 kind nm; do
[ -z "${ns2:-}" ] && continue
if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
warn " Left running; its volume hard-detaches at poweroff and rebuilds on restart."
skipped_delete_policy+=("$ns2/$nm")
continue
fi
if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
"${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
&& printf ' scaled 0 %s/%s %s\n' "$ns2" "$kind" "$nm" \
|| warn "could not scale $ns2/$kind/$nm"
done < <(pvc_workloads)
for line in "${deferred[@]:-}"; do
[ -z "${line:-}" ] && continue
# shellcheck disable=SC2086
set -- $line
"${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
&& printf ' scaled 0 %s/%s %s (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
done
# 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
# rebuilds; this is what we are buying with all of the above.
wait_for "UniFi VMI gone" \
'[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
wait_for "all Postgres instances stopped" \
'[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
wait_for "all Longhorn volumes detached" \
'[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
echo
warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
printf ' %s\n' "${skipped_delete_policy[@]}"
warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
fi
echo
ok "QUIESCED. Safe to power off."
cat <<'NEXT'
Next, from inside the devcontainer (it holds the become password):
cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
-m shell -a 'shutdown -h +1 "maintenance"'
THEN, in this order and not before:
1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
2. Router / switch last.
NEXT
}
# ---------------------------------------------------------------- resume
do_resume() {
need kubectl
# THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
# clears it, so resuming Flux alone leaves all three databases down and every app that needs
# one looking broken for reasons that point nowhere near Postgres.
log "un-hibernating CloudNativePG clusters"
local c ns name
for c in "${CNPG[@]}"; do
ns="${c%%/*}"; name="${c##*/}"
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
done
# Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
# undo the scale-downs by hand.
log "resuming Flux Kustomizations"
local k
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
done
ok "all Kustomizations resumed"
wait_for "Flux fully reconciled" \
'[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
echo
ok "RESUMED."
cat <<'NEXT'
Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
NEXT
}
do_status() {
need kubectl
printf 'nodes: %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
printf 'flux: %s/%s Ready\n' \
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
printf 'longhorn: %s attached, %s detached, %s not healthy\n' \
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
printf 'postgres: %s instance pod(s)\n' \
"$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
printf 'unifi VM: %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
}
case "${1:-}" in
quiesce) do_quiesce ;;
resume) do_resume ;;
status) do_status ;;
*) cat >&2 <<USAGE
usage: $0 {quiesce|resume|status}
quiesce suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
PVC-holding workload, and wait for all Longhorn volumes to detach.
Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
resume the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
status one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
Run it inside the pinned toolbox, which supplies kubectl and the context:
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
USAGE
exit 2 ;;
esac
+24
View File
@@ -127,6 +127,30 @@ resources:
# so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
# cannot disagree about the domain.
- ../platform/keycloak/client-urls.yaml
# PASSKEYS — the same one-shot failure, for the WebAuthn passwordless policy and the
# browser login flow. Measured on prod 2026-10-18, hours after the passkey declaration
# merged: the KeycloakRealmImport read Done=True / Started=False with the whole
# webAuthnPolicyPasswordless* block and the restructured browser-mfa flow INERT. This
# CronJob converges the policy and the enrolment required action, and builds the
# declared browser flow BLUE/GREEN: under a new alias, verified by readback, and
# structurally unable to write the flow that is currently bound (a half-rewritten bound
# flow is a total SSO outage, and this repo has had one). It BINDS the verified flow
# (one atomic PUT of browserFlow) ONLY when passkey-desired-state.yaml sets
# bind-browser-flow: "true"; the default "false" leaves the realm's binding untouched.
# See the script header in that file for the full rationale.
- ../platform/keycloak/passkey-reconciler.yaml
# Its DESIRED STATE, GENERATED from keycloak-realm-import.yaml by
# ci/render-keycloak-passkey-policy.py and gated by the L0 check. Same reason as
# client-urls.yaml above: the reconciler pod has no Kubernetes API access
# (automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR
# and its input must be MOUNTED.
#
# IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above, for exactly
# the reason given for client-urls.yaml: webAuthnPolicyPasswordlessRpId is the
# ${BASE_DOMAIN} token, and a split across Kustomizations would let one token resolve to
# two different values. An RP ID is baked into every passkey at enrolment and can never
# be re-keyed, so a domain the two files disagree about is an unrecoverable credential.
- ../platform/keycloak/passkey-desired-state.yaml
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
- ../platform/keycloak/httproute.yaml
- ../platform/keycloak/servicemonitor.yaml
+112
View File
@@ -0,0 +1,112 @@
---
# =================================================================================================
# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
# =================================================================================================
# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
# because `1/1 Ready` looks perfectly healthy.
#
# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
# reboots one node at a time on a schedule, which turns a latent single point of failure into a
# recurring one, unattended, in the middle of the night.
#
# -------------------------------------------------------------------------------------------------
# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
# -------------------------------------------------------------------------------------------------
# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
# deploy controller from a bundled manifest, not by Flux:
#
# $ kubectl -n kube-system get addon coredns
# NAME SOURCE CHECKSUM
# coredns /data/k3s/server/manifests/coredns.yaml 8cd91595...
# $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
# deploy@k3s-prod-1 k3s
#
# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
#
# The two durable alternatives were weighed and rejected:
# * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
# a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
# of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
# would have to be refreshed by hand forever. Too high a price for a replica count.
# * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
# controller would both own one object and fight continuously. That is the shape of the
# `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
# both owning one resource, ~2500 events per 10 minutes).
#
# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
# and scales back to 2 within one reconcile (~15s by default).
#
# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
#
# -------------------------------------------------------------------------------------------------
# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
# -------------------------------------------------------------------------------------------------
# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
# metric to be a valid object, not because coredns is expected to scale on load: measured via
# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
#
# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
#
# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
# never balloon DNS pods across the cluster.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: coredns
# kube-system, because that is where k3s puts coredns. This is the only object this component
# places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
# mutates only the coredns Deployment's scale subresource.
namespace: kube-system
labels:
app.kubernetes.io/name: coredns
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: coredns
minReplicas: 2
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
# Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
averageUtilization: 80
behavior:
scaleDown:
# 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
# queries and every client's negative cache. The HPA should be reluctant in this direction.
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 600
scaleUp:
# Prompt in the direction that restores redundancy — this is the path taken after a k3s
# restart has re-staged `replicas: 1`.
stabilizationWindowSeconds: 0
policies:
- type: Pods
value: 2
periodSeconds: 60
+4
View File
@@ -7,3 +7,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- coredns-custom.yaml
# HA floor for the coredns Deployment itself — a separate concern from the split-horizon server
# block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via
# its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning.
- hpa.yaml
+206
View File
@@ -31,6 +31,8 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
| `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
| `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
| `passkey-reconciler.yaml` (CronJob: WebAuthn passwordless policy + enrolment action + the **browser flow** ← git every 15m; the flow blue/green, never mutating the bound one, **bound only when git says so**) | **platform-config** | config |
| `passkey-desired-state.yaml` (**GENERATED** ConfigMap: the passkey desired state — plus the one hand-edited line, `bind-browser-flow`) | **platform-config** | config |
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
@@ -154,6 +156,210 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
a realm edit landed.
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
comments, not repeated here. The headline choices:
| Decision | Value | One-line reason |
|---|---|---|
| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
### The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)
Same one-shot failure this file already documents twice above, third field over. Measured on prod
2026-10-17:
```
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
{"status":"False","type":"HasErrors"}]}
```
`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
and three new sub-flows added to this file change nothing about how anyone logs in. The
client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
mapper or a changed scope still needs the realm import to actually re-run" case named above.
`passkey-reconciler.yaml` closes that gap the same way the client reconciler closed the
secret and URL halves: a CronJob on `*/15` that authenticates to the admin API and converges
declared state. It is a **sibling**, not a second pass of the client reconciler, precisely so that
reconciler's allow-list — and its guarantee that prod's real user and the `media-users` /
`platform-admins` / `viewers` groups are **structurally unreachable** — stays literally true.
Its desired state is `passkey-desired-state.yaml`, **generated** from the realm import by
`ci/render-keycloak-passkey-policy.py` and gated by
`conformance/tests/l0-static/keycloak-passkey-reconciler`, so the RP ID is declared exactly once.
That matters more here than for client URLs: an RP ID is baked into every credential at enrolment
and **can never be re-keyed**, so two declarations free to skew about the domain would be an
unrecoverable-credential bug, not a cosmetic one.
| Declared thing | Who applies it | Why |
|---|---|---|
| `webAuthnPolicyPasswordless*` (realm policy) | **The reconciler. GitOps.** | Realm top-level attributes; strict read-modify-write of `GET`/`PUT /admin/realms/homelab` |
| `webauthn-register-passwordless` required action | **The reconciler. GitOps.** | Per-alias `PUT`, one literally-named alias; enabled, `defaultAction: false` |
| The `browser-mfa` flow restructure | **The reconciler builds and verifies it. Git decides whether it is bound.** | Blue/green: build under a new alias, verify, and flip `browserFlow` in one atomic PUT **only when `bind-browser-flow: "true"`** — see below |
#### The flow converges blue/green — it never mutates the flow you log in through
Converging a five-flow / ten-execution restructure needs create-subflow,
create-execution, set-requirement and bind-config as **separate** admin API calls, with no
transaction. Doing that to the flow bound as `browserFlow` would mean a run dying part-way
leaves a **half-built login path** for Grafana, Gitea, Nextcloud, Headlamp, Semaphore,
PiHole, Crafty, the *arr stack *and* kubectl. This file already documents a total SSO
outage from a realm change, and the conditional-OTP sub-flow — which the declared
structure moves **one level deeper** than it sits today — once 403'd every platform-admin.
That risk is real but it is **not inherent**: it follows only from mutating the *bound*
flow. So the reconciler does not.
1. **Build** the declared tree under a **new, unused alias** — the declared alias plus a
hash of the desired shape, applied to every flow in the tree. Every multi-call write
lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound
flow and **zero user-visible effect**.
2. **Verify** by readback — nesting levels, ordering, authenticators, requirements, and
the conditional-OTP config *contents*. Nothing is bound unless this passes, so the
regression that once 403'd every platform-admin is caught **before** it can bite.
3. **Bind** with **one atomic PUT** of the realm, setting `browserFlow` — **only if git says
so** (next section). Otherwise the run stops here and logs `WOULD BIND browser-mfa-<hash>`.
4. **Rollback is a PR plus one write** (below), and nothing is ever deleted, so the old flow
stays usable.
#### The bind switch: `bind-browser-flow` in `passkey-desired-state.yaml`
That generated file carries exactly one hand-edited line, `bind-browser-flow`, and it is
`"false"` by default. The renderer carries it through on `--write`, and `--check` refuses
anything but the quoted strings `"true"` / `"false"` (an unquoted `true` is a YAML boolean,
which a ConfigMap rejects at apply time and which would fail the whole `platform-config`
Kustomization). The CronJob reads it from the same mount as the desired state; no flag can
widen it.
| `bind-browser-flow` | What the reconciler does to the realm's `browserFlow` |
|---|---|
| `"false"` (default) | **Never writes it.** It builds and verifies the candidate (or reuses an already verified one), logs `WOULD BIND <alias>`, and exits 0. Whatever the realm is bound to — including your rollback — is left alone on every run. A hand-edited bound flow is left alone too (no rebuild beside it). |
| `"true"` | Binds the verified candidate in one atomic PUT, **and keeps asserting it**: a binding changed by hand is changed back within 15 minutes, and a hand-edit *inside* the bound flow makes the next run build and bind a fresh copy. That is GitOps doing its job — so never hotfix the flow while this is `"true"`. |
**Merging with `"false"` changes nothing about how anyone logs in.** Passes 1 and 2 do write
the realm (the passwordless policy and the enrolment action, neither of which any login
step uses until a flow offers passkeys), and pass 3 creates the unbound candidate flow, but
`browserFlow` is untouched.
**Going live** is a later one-line PR — `bind-browser-flow: "true"` — made **after** the
candidate has been tested in a browser (see "Testing the candidate flow before binding it").
**Rolling back**, in this order:
1. A PR setting `bind-browser-flow: "false"`, merged and applied by Flux. **First** — while it
is `"true"`, the next run rebinds the candidate over your rollback.
2. One write: bind the previous flow again. The Job log of the bind printed it
(`ROLLBACK is one write: set browserFlow back to ...`); it is `browser-mfa` for the first
go-live. Admin console: *Authentication → Flows → <previous flow> → Action → Bind flow →
Browser flow*, or `PUT /admin/realms/homelab` as a read-modify-write of the full realm
representation (a partial body drops the passkey policy).
If login is broken *right now*, do step 2 immediately and step 1 straight after: the worst
case of that order is that one scheduled run (≤15 min) rebinds the candidate before Flux
applies the PR, and you repeat step 2.
**The safety property is structural, not intentional.** The reconciler is *incapable* of
writing to the bound flow or anything beneath it: the live `browserFlow` alias and its
whole subtree of sub-flow aliases and execution ids are read first and screened against
every write before a socket opens. It is **fail-closed** — until that binding has been
read, every flow write is refused, and if the bound flow's executions cannot be read the
flow pass writes nothing at all. There is **no `DELETE`** anywhere in its allow-list.
**Idempotence is load-bearing**, not a nicety: a rebuild-and-reflip every 15 minutes
would churn the login path on a schedule. The bound flow is compared structurally against
the declared shape and, when they match, the run issues **zero writes and builds
nothing**. Sub-flow and config *aliases* are deliberately **not** compared — a flow the
reconciler built carries a hash suffix, so comparing labels would make it permanently
unrecognisable as converged.
**Recovery.** A crashed run can leave a partial flow under the primary candidate alias. A
later run reads it: if it matches desired it is a completed-but-unbound build and is
simply **bound**; if not, it is left alone and the next candidate alias is used. The
candidate count is **bounded**, so a persistently failing build fails loudly naming the
flows to clean up rather than creating unbounded sprawl.
```sh
# What the reconciler currently says. A normal run prints the full comparison.
kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200
# Which candidate it would bind (bind-browser-flow "false"):
kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300 \
| grep -E 'WOULD BIND|BOUND browserFlow|PROBLEM'
# Force a run now rather than wait for the tick. It does exactly what the CronJob does,
# including honouring bind-browser-flow. `kubectl create job --from=cronjob/...` CANNOT
# override the container command, so --audit / --no-bind need a one-off Job with the flag
# appended (--no-bind can only NARROW git's switch, never widen it).
kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
passkey-run-once
kubectl --context k3s-prod -n keycloak logs job/passkey-run-once
# ROLLBACK — see "The bind switch" above: PR bind-browser-flow "false" FIRST, then rebind
# the previous flow (Authentication -> Flows -> <previous> -> Action -> Bind flow ->
# Browser flow). The old flow is never deleted, so this always works.
# Regenerate the desired state after editing the passkey policy or the flow in the realm
# import (the merge gate will demand it).
python3 ci/render-keycloak-passkey-policy.py --write
```
### Testing the candidate flow before binding it
With `bind-browser-flow: "false"` the candidate exists but nothing logs in through it. Test it
on **one client** without touching the realm binding, using a per-client flow override:
1. Find the candidate: the Job log line `WOULD BIND browser-mfa-<hash>`.
2. Admin console → *Clients → `account-console` → Advanced → Authentication flow overrides →
Browser Flow* = `browser-mfa-<hash>`. This is a temporary, hand-made client change; remove it
in step 7.
3. **Password fallback.** Private window → `https://keycloak.<base domain>/realms/homelab/account`.
Expect a **username** screen, then a choice: passkey, or the password form. As a user with no
passkey, choose password and sign in.
4. **Platform-admin TOTP on the password branch.** Repeat step 3 as a `platform-admins` member:
after the password, a TOTP prompt **must** appear. If it does not — or you get a 403 — the
conditional-OTP step is wrong: stop, do not bind.
5. **Passkey enrolment.** Signed in to the account console → *Signing in → Passkey → Set up*. A
browser/OS passkey prompt must appear; if it does not, the RP ID is wrong for the origin and
nothing server-side will say so.
6. **Passkey login.** Sign out; private window again; username, then choose passkey. Expect to be
signed in with **no** TOTP prompt, even as a platform-admin.
7. Remove the override from step 2.
8. Only now: the one-line PR `bind-browser-flow: "true"`. The next run binds the candidate you
tested (same alias — the hash is of the declared shape).
### Verifying it actually works (nobody has, as of 2026-10-29)
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
it on the BOUND flow, after go-live, in order:
```sh
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
# Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
# -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
# is wrong for the origin and NOTHING will say so.
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
# prompt. A platform-admin must NOT be asked for TOTP on this path.
# 4. regression (the important one): a user with NO passkey must still get the password screen,
# and a platform-admin must still be asked for TOTP there.
```
A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
## Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
+285 -8
View File
@@ -74,6 +74,142 @@ spec:
bruteForceProtected: true
permanentLockout: false
sslRequired: external
# ===========================================================================================
# PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
# ===========================================================================================
# Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
# what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
# WebAuthn policies per realm and they are not interchangeable:
#
# webAuthnPolicy* -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
# authenticator (a security key presented AFTER a password).
# webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
# `webauthn-authenticator-passwordless` (a passkey presented
# INSTEAD of a password). This is the one passkeys use.
#
# Setting the wrong family is a silent no-op: the passwordless authenticator reads
# getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
# so a policy written into the two-factor fields is simply never consulted by a passkey login.
# Only the passwordless family is declared here; the two-factor family is deliberately left at
# Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
# (auth-otp-form) is the second factor for platform-admins, see the flow section.
#
# ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
# -------------------------------------------------------------
# Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
# This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
# A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
# lock-out from the whole platform AND from the cluster API in one step — and the only repair
# paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
# expected gain from forcing enrolment (one human stops typing a password) is small; the
# expected loss is the platform. So: additive, optional, and password login keeps working for
# every user who never enrols.
#
# Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
# long note in the flow section for why declaring one would be actively DANGEROUS), and the
# passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
#
# RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
# ------------------------------------------------------------------------------
# The RP ID is baked into every credential at enrolment and can never be changed afterwards:
# a passkey created under one RP ID is unusable under another, with no error the user can act
# on — the browser simply reports that no credential is available. It is also browser-enforced,
# so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
# registrable-domain suffix of, the origin's effective domain. Keycloak is served at
# keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
# are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
# the full URL and NOT a scheme — "https://..." here would fail every ceremony.
#
# We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
# reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
# bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
# on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
# hostname; and the client-URL reconciler exists precisely because moving a hostname in git
# silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
# Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
# every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
# future WebAuthn relying party under this base domain could assert the same RP ID. In a
# homelab where every such party is already behind THIS IdP that is not a trust boundary worth
# buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
# passkey, which is a re-enrolment, not a repair.
#
# Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
# value, as every hostname in this file — there is no second place the domain is decided.
webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
# The human-readable name the browser/OS shows in the passkey prompt and stores beside the
# credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
# OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
# updateRealm applies the passwordless policy ONLY when this field is non-empty
# (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
# whole policy silently.
webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
# ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
# inherited so the realm's crypto is auditable from git. ES256 is what every platform
# authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
# RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
# deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
# that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
# moment to discover it. Order is a preference list, not an allow-list ordering requirement.
webAuthnPolicyPasswordlessSignatureAlgorithms:
- ES256
- RS256
# DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
# theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
# authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
# This is not optional for passkeys — a non-discoverable credential cannot be found by the
# browser without the server first naming it, so it is a second factor, not a passkey.
# (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
# residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
# residentKey="required", so discoverable credentials are genuinely requested — but it also
# means this policy cannot express residentKey:"preferred" at all.)
webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
# USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
# say so in the assertion. This is the field that makes a passkey login at least as strong as
# the password it replaces: with verification merely "preferred", possession of an unlocked
# phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
# single-factor possession check, strictly weaker than the password+TOTP path beside it.
# Required makes the passkey branch inherently two-factor (something you have + something you
# are/know), which is the premise the flow section relies on when it does NOT ask a
# passkey-authenticated platform-admin for TOTP as well.
webAuthnPolicyPasswordlessUserVerificationRequirement: required
# "not specified" is Keycloak's sentinel for "either kind of authenticator"
# (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
# entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
# every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
# reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
# The operator should be able to enrol a laptop AND a phone AND a hardware key.
webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
# ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
# no trust anchors to validate here (no metadata service is configured) and nothing to compare
# it against, since the AAGUID allow-list below is deliberately empty — so the statement would
# be collected, unverifiable, and would add an extra "share model information?" consent prompt
# on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
# we do not have one, so asking for it is cost without benefit.
webAuthnPolicyPasswordlessAttestationConveyancePreference: none
# Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
# would have to be maintained by hand, and every new phone, laptop or key the operator buys
# would be rejected at enrolment with a message that does not explain itself. It is also moot
# while attestation is "none" (above) — with no attestation statement there is no trustworthy
# AAGUID to match against in the first place.
webAuthnPolicyPasswordlessAcceptableAaguids: []
# Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
# let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
# path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
# phone, scans it, and completes a biometric check — which routinely takes over a minute. A
# 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
# this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
webAuthnPolicyPasswordlessCreateTimeout: 300
# Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
# authenticator the user already registered is refused by the authenticator itself instead of
# producing a second, indistinguishable credential in the account console. Cheap, and it keeps
# "which passkey is which" answerable — which matters when the recovery story is "delete the
# lost one".
webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
# webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
# origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
# a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
# origins is only ever a widening of who may ask for these credentials.
# --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
roles:
realm:
@@ -528,14 +664,105 @@ spec:
"access.token.claim": "true"
"userinfo.token.claim": "true"
"claim.name": groups
# --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
# platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
# Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
# the whole import — if so, the fallback is to comment this block + browserFlow and configure
# the conditional-OTP-by-role flow via the admin console. ---
# ===========================================================================================
# MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
# bound to browser-mfa at the bottom of this file.
# ===========================================================================================
# Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
# (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
# this block + browserFlow and configure the flow via the admin console.
#
# THE SHAPE, AND WHY IT IS THIS SHAPE
# -----------------------------------
# browser-mfa (top level, bound as browserFlow)
# auth-cookie ALTERNATIVE already-authenticated SSO cookie
# identity-provider-redirector ALTERNATIVE (no IdPs configured; kept for parity)
# browser-mfa-forms ALTERNATIVE the interactive path:
# auth-username-form REQUIRED WHO are you (no password yet)
# browser-mfa-passkey-or-password REQUIRED ...and PROVE it, one of two ways:
# webauthn-authenticator-passwordless ALTERNATIVE a passkey, or
# browser-mfa-password-and-otp ALTERNATIVE the password path:
# auth-password-form REQUIRED password, then
# browser-mfa-conditional-otp CONDITIONAL TOTP iff platform-admin
#
# WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
# `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
# behind that: the server must know WHICH user is logging in before it can decide whether a
# passkey is even available, and the combined form has already taken the password by the time
# it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
# same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
# EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
# one. That cost is accepted deliberately — it is the enabling change, it is what Google and
# Microsoft logins already do, and the alternative (below) is worse.
#
# WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
# does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
# authenticator can legally run as a first step with no user in context. We deliberately do not
# do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
# "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
# with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
# the authenticator. A passkey execution placed before the username form is therefore offered
# to EVERYONE, so every login — including every password-only user and every family member who
# will never own a passkey — would open on a passkey prompt they must dismiss. That is the
# opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
# context, which is what makes the graceful degradation below actually work.
#
# WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
# claim rests on). With a user in context, the engine takes the non-requiresUser branch and
# raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
# authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
# .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
# conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
# marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
# no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
# the passkey screen and keeps the password as a "Try another way" option. Note that for a
# user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
# selection list (the user's own credential ordering), not by the priorities below.
#
# WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
# PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
# for TOTP. That is intentional, and it rests on the policy above: userVerification is
# `required`, so the authenticator has already proved possession of hardware AND verified the
# human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
# top would add a replayable, phishable shared secret to a credential that is neither, for
# friction on every login. If a future operator disagrees, the change is small and local: move
# the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
# browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
#
# WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
# --------------------------------------------------------------------
# Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
# move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
# requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
# `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
# exactly those plus delete_account — the `else` branch that installs the full default set
# (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
# backstop either: it fires only when the realm has ZERO required-action providers.
#
# So declaring one action would create this realm with that action and delete_account and
# NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
# no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
# with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
# enrol one, while auth-otp-form below still demands it — a hard lock-out of every
# platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
# change whose stated purpose was to ADD a login method.
#
# And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
# DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
# defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
# in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
# additive-and-optional posture this change wants: the action exists and can be triggered for
# a user on demand, but it is not forced on anyone at next login. Declaring it would restate
# the default while deleting ten siblings.
#
# (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
# block is safe: importAuthenticationFlows adds the declared flows and then backfills each
# unbound binding — browser, registration, direct grant, reset credentials, client auth, first
# broker login, docker auth — from the built-in defaults individually. Declaring flows does not
# suppress the built-ins; declaring required actions does suppress them.)
authenticationFlows:
- alias: browser-mfa
description: "Browser flow with conditional OTP required for platform-admins."
description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
providerId: basic-flow
topLevel: true
builtIn: false
@@ -555,13 +782,63 @@ spec:
priority: 30
flowAlias: browser-mfa-forms
userSetupAllowed: false
# Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
# username is never optional, and neither is proving possession of the account.
- alias: browser-mfa-forms
description: "Username/password then conditional OTP."
description: "Username, then passkey or password (+ conditional OTP)."
providerId: basic-flow
topLevel: false
builtIn: false
authenticationExecutions:
- authenticator: auth-username-password-form
# auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
# user WITHOUT taking a password, which is the prerequisite for offering a passkey.
# Replaces the previous auth-username-password-form; see the header for the UX cost.
- authenticator: auth-username-form
authenticatorFlow: false
requirement: REQUIRED
priority: 10
userSetupAllowed: false
- authenticatorFlow: true
requirement: REQUIRED
priority: 20
flowAlias: browser-mfa-passkey-or-password
userSetupAllowed: false
# THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
# passkey first for users who have one; users who do not fall through to the password branch
# automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
- alias: browser-mfa-passkey-or-password
description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
providerId: basic-flow
topLevel: false
builtIn: false
authenticationExecutions:
# webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
# (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
# declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
# REQUIRED it would force a passkey on every user, which is the lock-out this change is
# explicitly designed not to create.
- authenticator: webauthn-authenticator-passwordless
authenticatorFlow: false
requirement: ALTERNATIVE
priority: 10
userSetupAllowed: false
- authenticatorFlow: true
requirement: ALTERNATIVE
priority: 20
flowAlias: browser-mfa-password-and-otp
userSetupAllowed: false
# The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
# then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
# changed; it simply no longer sits on the path a passkey login takes.
- alias: browser-mfa-password-and-otp
description: "Password, then conditional OTP for platform-admins."
providerId: basic-flow
topLevel: false
builtIn: false
authenticationExecutions:
# auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
# half of the old combined form, now that the username has already been collected.
- authenticator: auth-password-form
authenticatorFlow: false
requirement: REQUIRED
priority: 10
@@ -0,0 +1,189 @@
---
# =============================================================================
# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
#
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
# (runs this renderer with --check on every merge)
#
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
# the WebAuthn passwordless policy, the enrolment required action, and the declared
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
# it and (a) converges the policy and the required action onto it, (b) builds the
# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
# writing the flow users are logging in through, and (c) BINDS that verified flow as
# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
#
# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
# operator decision, carried through by the renderer from this committed file (see
# ci/render-keycloak-passkey-policy.py). It defaults to "false".
#
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
# once Done=True, so a realm-level field or a flow edited in git never reaches the
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
# the level-triggered loop that closes that; this file is what it reconciles
# TOWARDS.
#
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
# user-verification requirement and the flow tree would be a second source of truth,
# free to disagree with the realm import — the same "two declarations that can
# disagree" shape as the bug being closed. It is rendered from the import and the
# render is gated, so the two cannot drift.
#
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
# substitution token exactly as the realm import writes it. Both files are rendered
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
# same value in both and the domain is decided in exactly one place.
#
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
# nine more (a hard lock-out of every platform-admin). It is derived instead from
# the presence of a webauthn-authenticator-passwordless execution in the flow:
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
# false, so enrolment stays opt-in.
#
# A field NOT declared on the realm in the import is simply ABSENT here, and the
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: keycloak-passkey-desired-state
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak-passkey-desired-state
app.kubernetes.io/part-of: keycloak
data:
# ---------------------------------------------------------------------------
# THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
# "false", QUOTED.
#
# "false" the reconciler builds and verifies the candidate browser flow and
# logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
# Whatever the realm is bound to -- including a rollback -- is left alone.
# "true" it binds the verified candidate in one atomic PUT, and keeps the
# realm bound to it: a binding changed by hand is changed back.
#
# GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
# "Passkeys"), then a PR setting this to "true".
# ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
# one write setting browserFlow back to the previous alias from the Job log.
# ---------------------------------------------------------------------------
bind-browser-flow: "false"
passkey-desired-state.json: |
{
"authenticatorConfig": {
"conditional-otp-platform-admin": {
"condUserRole": "platform-admin",
"negate": "false"
}
},
"browserFlow": "browser-mfa",
"flows": {
"browser-mfa": {
"executions": [
{
"kind": "authenticator",
"name": "auth-cookie",
"requirement": "ALTERNATIVE"
},
{
"kind": "authenticator",
"name": "identity-provider-redirector",
"requirement": "ALTERNATIVE"
},
{
"kind": "flow",
"name": "browser-mfa-forms",
"requirement": "ALTERNATIVE"
}
],
"topLevel": true
},
"browser-mfa-conditional-otp": {
"executions": [
{
"configAlias": "conditional-otp-platform-admin",
"kind": "authenticator",
"name": "conditional-user-role",
"requirement": "REQUIRED"
},
{
"kind": "authenticator",
"name": "auth-otp-form",
"requirement": "REQUIRED"
}
],
"topLevel": false
},
"browser-mfa-forms": {
"executions": [
{
"kind": "authenticator",
"name": "auth-username-form",
"requirement": "REQUIRED"
},
{
"kind": "flow",
"name": "browser-mfa-passkey-or-password",
"requirement": "REQUIRED"
}
],
"topLevel": false
},
"browser-mfa-passkey-or-password": {
"executions": [
{
"kind": "authenticator",
"name": "webauthn-authenticator-passwordless",
"requirement": "ALTERNATIVE"
},
{
"kind": "flow",
"name": "browser-mfa-password-and-otp",
"requirement": "ALTERNATIVE"
}
],
"topLevel": false
},
"browser-mfa-password-and-otp": {
"executions": [
{
"kind": "authenticator",
"name": "auth-password-form",
"requirement": "REQUIRED"
},
{
"kind": "flow",
"name": "browser-mfa-conditional-otp",
"requirement": "CONDITIONAL"
}
],
"topLevel": false
}
},
"realm": "homelab",
"requiredAction": {
"alias": "webauthn-register-passwordless",
"defaultAction": false,
"enabled": true
},
"webAuthnPolicyPasswordless": {
"webAuthnPolicyPasswordlessAcceptableAaguids": [],
"webAuthnPolicyPasswordlessAttestationConveyancePreference": "none",
"webAuthnPolicyPasswordlessAuthenticatorAttachment": "not specified",
"webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": true,
"webAuthnPolicyPasswordlessCreateTimeout": 300,
"webAuthnPolicyPasswordlessRequireResidentKey": "Yes",
"webAuthnPolicyPasswordlessRpEntityName": "Homelab SSO",
"webAuthnPolicyPasswordlessRpId": "${BASE_DOMAIN}",
"webAuthnPolicyPasswordlessSignatureAlgorithms": [
"ES256",
"RS256"
],
"webAuthnPolicyPasswordlessUserVerificationRequirement": "required"
}
}
File diff suppressed because it is too large. Load diff
+30
View File
@@ -1445,6 +1445,36 @@ spec:
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
that would catch the healer being quietly absent during a domain migration.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
# --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
# The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
# enrolment required action, and the browser login flow (blue/green; bound only when
# git's bind-browser-flow is "true"). There is no failure that is "expected by
# design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
# run. So a failed Job is always news — drift it may not repair (e.g. a missing
# CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
# verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
# backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
# single transient failure while still firing within about two ticks.
# A separate rule from the client reconciler's on purpose: different CronJob,
# different job_name, different runbook — one cause, one alert.
# UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
# did not exist yet. The metric and label are the ones the sibling rule above uses.
- alert: KeycloakPasskeyReconcilerFailing
expr: |
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Keycloak passkey reconciler failing"
description: >-
A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
policy, the passkey enrolment action or the browser login flow may differ from
git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
never writes the flow users are logging in through, so a failure here does NOT
by itself mean login is broken. Read the Job log: every PROBLEM line names the
pass and the exact divergence.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
# =====================================================================================
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
@@ -0,0 +1,90 @@
# promtool rule-unit tests for KeycloakPasskeyReconcilerFailing (platform/monitoring/alerts.yaml,
# group platform-health.identity).
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
# conformance/tests/l0-static/alert-rules-unit-tested)
#
# WHAT THE RULE CLAIMS, PINNED IN BOTH DIRECTIONS. The passkey reconciler
# (platform/keycloak/passkey-reconciler.yaml) runs every 15 minutes with backoffLimit 0, so one
# failed tick is one Job with kube_job_status_failed = 1, kept for an hour (ttlSecondsAfterFinished)
# and then garbage-collected. Its steady state is GREEN whether or not git lets it bind the browser
# flow, so a failed Job is always news. What can go wrong with the rule:
# * it never fires — a job_name regex that misses the CronJob's Job names (`<cronjob>-<minutes>`);
# * it fires on the SIBLING — the client-secret reconciler lives in the same namespace, and its
# failures already have their own alert and runbook; one cause, one alert;
# * it fires on a single transient failure that the next tick cleared — `for: 15m` exists for that.
# Fixture names are prod's real shape: CronJob Jobs are named <cronjob>-<scheduled unix minutes>.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
# =============================================================================================
# 1. HEALTHY: every passkey Job succeeded -> SILENT. The series exist with failed = 0, so a rule
# that forgot the `> 0` would fire here.
# =============================================================================================
- interval: 1m
name: passkey reconciler succeeding - quiet
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
values: '0+0x60'
- series: 'kube_job_status_succeeded{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
values: '1+0x60'
alert_rule_test:
- eval_time: 60m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
# =============================================================================================
# 2. THE FAILURE: a passkey Job failed and stays failed -> FIRES after 15m (and not before).
# =============================================================================================
- interval: 1m
name: passkey reconciler Job failed - fires after 15m
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
values: '1+0x60'
alert_rule_test:
- eval_time: 10m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
- eval_time: 20m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts:
- exp_labels:
severity: warning
exp_annotations:
summary: "Keycloak passkey reconciler failing"
description: "A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey policy, the passkey enrolment action or the browser login flow may differ from git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler never writes the flow users are logging in through, so a failure here does NOT by itself mean login is broken. Read the Job log: every PROBLEM line names the pass and the exact divergence."
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
# =============================================================================================
# 3. THE SIBLING'S FAILURE IS NOT THIS ALERT. The client-secret reconciler's Job fails in the
# same namespace; KeycloakClientSecretReconcilerFailing owns it. This one must stay quiet.
# =============================================================================================
- interval: 1m
name: only the client-secret reconciler failing - passkey alert quiet
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-client-secret-reconciler-29889015"}'
values: '1+0x60'
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
values: '0+0x60'
alert_rule_test:
- eval_time: 60m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
# =============================================================================================
# 4. ONE TRANSIENT FAILURE, cleared by the next tick: the failed Job is garbage-collected (its
# series ends) after 5m here and the next Job succeeds -> never held for 15m -> QUIET.
# =============================================================================================
- interval: 1m
name: single transient failure cleared within 15m - quiet
input_series:
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889030"}'
values: '1+0x5'
- series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889045"}'
values: '_x15 0+0x45'
alert_rule_test:
- eval_time: 30m
alertname: KeycloakPasskeyReconcilerFailing
exp_alerts: []
-99
View File
@@ -1,99 +0,0 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
"extends": [
"config:recommended",
":dependencyDashboard",
":semanticCommits",
"helpers:pinGitHubActionDigests"
],
"timezone": "America/New_York",
"schedule": ["after 3am and before 6am"],
"prConcurrentLimit": 5,
"prHourlyLimit": 2,
"commitMessagePrefix": "chore(deps):",
"labels": ["dependencies", "automated-update"],
"ignoreUnstable": true,
"respectLatest": true,
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
"flux": {
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
},
"kustomize": {
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
},
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
"kubernetes": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)platform-config/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/",
"/(^|/)catalog/.+\\.ya?ml$/",
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
"/(^|/)infra/agent/.+\\.ya?ml$/"
]
},
"customManagers": [
{
"customType": "regex",
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/",
"/(^|/)versions\\.env$/"
],
"matchStrings": [
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
],
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
}
],
"packageRules": [
{
"description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
"matchUpdateTypes": ["minor", "patch", "digest"],
"automerge": true,
"automergeType": "pr",
"platformAutomerge": true
},
{
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
"matchUpdateTypes": ["major"],
"automerge": false,
"addLabels": ["needs-review", "breaking"]
},
{
"description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
"matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
"groupName": "kubevirt-cdi"
},
{
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
"automerge": false,
"addLabels": ["ci-tooling"]
},
{
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
"matchPackageNames": [
"kured",
"longhorn",
"cilium",
"cloudnative-pg",
"kube-prometheus-stack"
],
"automerge": false,
"addLabels": ["needs-review", "blast-radius"]
}
],
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
"ignorePaths": [
"**/*.example",
"**/*.sops.yaml",
"conformance/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
"docker": { "enabled": true }
}
+187
View File
@@ -0,0 +1,187 @@
// =================================================================================================
// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
// =================================================================================================
// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
//
// ERROR: Found errors in configuration
// Invalid configuration option: _comment
// Invalid configuration option: _docker_comment
// Invalid configuration option: _ignorePaths_comment
// Invalid configuration option: _kubernetes_comment
// Invalid configuration option: _unmanageable_comment
//
// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
// comments. So the fix keeps every word of the original rationale, positioned, instead of
// flattening it into a `description` string.
//
// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
//
// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to
// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing
// here changes it.
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
// Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
// RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
// It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
// split that gave image tags to Flux image-automation is formally closed — see
// .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
// opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
// user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
// anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
// defines the cluster). image-automation has been commented out at
// platform/kustomization.yaml:52 since it was authored and the cluster holds zero
// ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
// CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
// fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
// line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
// sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
// .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
"extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
"timezone": "America/New_York",
"schedule": ["after 3am and before 6am"],
"prConcurrentLimit": 5,
"prHourlyLimit": 2,
"commitMessagePrefix": "chore(deps):",
"labels": ["dependencies", "automated-update"],
"ignoreUnstable": true,
"respectLatest": true,
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
"flux": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/"
]
},
"kustomize": {
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/"
]
},
// ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
// manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
// this block is as broad as it is. Scope covers every place this repo writes a container image:
// platform components, the rendered app waves, the ServiceDefinition catalog that generates
// them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
// `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
// rendered output is reverted by the next render and would also fail ci/render-golden.sh.
// Renovate will raise BOTH, and both must land together.
"kubernetes": {
"managerFilePatterns": [
"/(^|/)platform/.+\\.ya?ml$/",
"/(^|/)platform-config/.+\\.ya?ml$/",
"/(^|/)apps/.+\\.ya?ml$/",
"/(^|/)catalog/.+\\.ya?ml$/",
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
"/(^|/)infra/agent/.+\\.ya?ml$/"
]
},
"customManagers": [
{
"customType": "regex",
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
"managerFilePatterns": [
"/(^|/)kustomization\\.ya?ml$/",
"/(^|/)versions\\.env$/"
],
"matchStrings": [
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
],
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
}
],
"packageRules": [
{
"description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
"matchUpdateTypes": [
"minor",
"patch",
"digest"
],
"automerge": true,
"automergeType": "pr",
"platformAutomerge": true
},
{
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
"matchUpdateTypes": [
"major"
],
"automerge": false,
"addLabels": [
"needs-review",
"breaking"
]
},
{
"description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
"matchDepNames": [
"kubevirt/kubevirt",
"kubevirt/containerized-data-importer"
],
"groupName": "kubevirt-cdi"
},
{
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
"matchFileNames": [
"ci/versions.env",
"infra/devcontainer/versions.env"
],
"automerge": false,
"addLabels": [
"ci-tooling"
]
},
{
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
"matchPackageNames": [
"kured",
"longhorn",
"cilium",
"cloudnative-pg",
"kube-prometheus-stack"
],
"automerge": false,
"addLabels": [
"needs-review",
"blast-radius"
]
}
],
// WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
// FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
// and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
// lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
// (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
// version before any automation can maintain them, and until then they drift on every pull with
// nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
// e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
// has been told not to leave; raising those ceilings is deliberate per-component work and is
// NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
// platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
// and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
// shape, not software to keep current. `conformance/**` was already ignored for the same
// reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
// have opened.
"ignorePaths": [
"**/*.example",
"**/*.sops.yaml",
"conformance/**",
"platform/image-automation/per-service/**",
"platform/kyverno/tests/**"
],
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
// over a running tag. That controller was never enabled and never will be here (see _comment),
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
"docker": {
"enabled": true
}
}
+2
View File
@@ -141,6 +141,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
every green signal missed.
- **`keycloak-passkey-reconciler`** — the passkey policy / enrolment action / browser-flow reconciler
failing. Never writes the flow you log in through; a missing `CONFIGURE_TOTP` is the cause that bites.
- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
`OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
rather than a 401, the browser follows it silently against a live SSO session, and the call
+162
View File
@@ -0,0 +1,162 @@
# cluster-maintenance-shutdown — powering the whole cluster off, and back on
**Operator procedure, not an alert page.** Nothing links here from an alert.
**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
page is for taking **all three nodes down at once**.
First written after the 2026-10-17 UPS move, which was improvised live because no procedure
existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
cluster, so it does nothing for these three baremetal nodes.
## Why you cannot just run `shutdown -h`
You mostly can now, and could not before.
Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.
**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
does not do:
1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
`running: false`; an eviction is a power cut to the guest.
2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
that Flux will neither set nor clear.
3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.
## The hazard that bit us — read this before scaling anything by hand
**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
to 0 then DELETES its PVCs and their Longhorn volumes.**
On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
misread as a miscount.
`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**
```bash
kubectl --context k3s-prod get statefulset -A \
-o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
```
A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
better outcome than a deleted one — take the rebuild.
## Shutting down
Run everything through the pinned toolbox; it supplies `kubectl` and the context.
```bash
# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
# powering off on top of a degraded volume is how one replica becomes zero.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
# PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
```
**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.
```bash
# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
./infra/devcontainer/dev.sh run bash -c \
'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
-m shell -a "shutdown -h +1 \"maintenance\""'
```
`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
`--list-hosts` if you want to see them first.
```bash
# 3. Confirm they are actually down before touching anything else.
for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
```
**Then, in this order and not before:**
| # | Machine | How |
|---|---|---|
| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
| 5 | **Router / switch** | Last. |
TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).
## Bringing it back
**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
2 of 3 for quorum, so start the nodes as a group rather than one at a time.
```bash
# 1. Nodes Ready, etcd quorum, storage reachable.
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'
# 2. Resume: un-hibernate Postgres, then resume Flux.
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
```
**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
exactly that reason.
Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
needs undoing by hand.
### What "still broken" looks like when it is actually fine
* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
`platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
`platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
artifact — if it is, wait.
* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
healthy on its own. Worry only if it is still degraded after ~30 minutes.
* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
`/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.
### What is actually broken and needs you
* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
**exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
Delete the pod once and it comes up clean:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
```
* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
recover; delete them.
## Verifying you are actually back
```bash
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI
# DNS, through PiHole's VIP
./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'
# The Gateway actually serving (401 on an auth-gated host is CORRECT)
for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
done
```
Expect `200`, `200`, `401`.
+69
View File
@@ -0,0 +1,69 @@
# keycloak-passkey-reconciler — the passkey reconciler's Job is failing
**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
**Expression:**
```
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
```
## What this means here
`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
at `Done=True`:
1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
(`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
`bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
default) a run that builds and verifies a candidate but does not bind it is a **success**.
So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
is structurally unable to write the flow users are currently logging in through, so **a failing Job
does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
`CONFIGURE_TOTP` required action (see below).
## First commands
Every command runs in the pinned toolbox — kubectl is not on the workstation:
```sh
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
# What git currently asks for (the bind switch):
grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
```
## Likely causes, most likely first
1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
(Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
but the readback differs (the log names the step). Login is unaffected. If it names the
conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
platform-admin, caught before it went live.
3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
`requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
lock-out for platform-admins: re-register the action in the admin console
(Authentication → Required actions). The reconciler cannot repair it.
4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
`keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
(`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
transient; if persistent, the realm is bound to a flow that no longer exists.
## When to escalate or stop
- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
`"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
alias printed in the Job log).
- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.
+8
View File
@@ -52,6 +52,13 @@ configMapGenerator:
- backup-failure=backup-failure.md
- backup-missed=backup-missed.md
- backup-partial=backup-partial.md
# Operator procedure, not an alert page — nothing links here from an alert. Served because
# the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
# virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
# improvised live, and the one hazard it exposed -- a StatefulSet with
# persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
# destroyed monitoring/loki's volume before anyone noticed.
- cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
- cdi-import-stuck=cdi-import-stuck.md
- cert-expiry=cert-expiry.md
- cert-issuance=cert-issuance.md
@@ -70,6 +77,7 @@ configMapGenerator:
- image-automation=image-automation.md
- image-policy-stalled=image-policy-stalled.md
- keycloak-client-secrets=keycloak-client-secrets.md
- keycloak-passkey-reconciler=keycloak-passkey-reconciler.md
- keycloak-db=keycloak-db.md
- keycloak-down=keycloak-down.md
- kubevirt-control-plane=kubevirt-control-plane.md