The Keycloak CR has no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the
server image; the three keycloak-k8s-resources release URLs in
platform/keycloak/kustomization.yaml are therefore THE Keycloak version, and Renovate
could not see them (no `# renovate:` marker). They now carry markers under one depName
(one branch, groupName keycloak-operator), with a packageRule: automerge false, labels
needs-review + blast-radius; majors still wait on the dashboard.
The untagged quay.io/keycloak/keycloak in catalog/keycloak.yaml renders
apps/wave-1/keycloak.yaml, which no kustomization lists. It is pinned to the operator's
26.0.6 as documentation, re-rendered, and Renovate is disabled for that image so it
cannot raise PRs against a file nothing deploys. The render is kept (not deleted): it is
the fixture for apps/wave-1/tests/app-slis.test.yaml's keycloak SLI rule tests.
README: an "Upgrading Keycloak" section — an operator bump must be checked against
passkey-reconciler.yaml's 26.0.6 Admin API assumptions.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator decision. Both are held at every update type and labelled needs-review:
- quay.io/oauth2-proxy/oauth2-proxy (render.py constant, rendered sidecars, crafty). Since v7.13.0
oauth2-proxy validates refreshed OIDC sessions against the access token, every sidecar uses
--cookie-refresh, and that path is untested against this Keycloak.
- ghcr.io/renovatebot/renovate (and renovatebot/github-action if ever reintroduced). Nothing
re-validates renovate.json5 against the new version before it takes effect.
The validator passes on 44.167.0. Evaluated with Renovate's own packageRules engine: exactly 2 of 29
branch rows change (automerge true -> false, +needs-review); the other 27 are identical.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Audit of the first completed dry runs (353/354) plus a debug-level local reproduction:
- App-image updates edited only the generated apps/wave-*. A ServiceDefinition has no apiVersion,
so the kubernetes manager extracts nothing from catalog/, and every such PR would fail
render-golden (reproduced: GOLDEN DRIFT). New regex managers for catalog source.image and
render.py OAUTH2_PROXY_IMAGE put the source and generated edits in one branch (render-golden
rc 0, reproduced).
- CNPG Cluster imageName was untracked (3 Clusters), while the grant Job was tracked. New manager.
- allowedVersions mirror catalog update.range: this drops gitea 28.x, qbittorrent 20.x, radarr 6,
nextcloud 35 and postgresql 18.
- Never auto-merge: gitea, act_runner, postgresql (blast radius), flux range widening, python
minors. Majors wait for a Dependency Dashboard tick.
- velero plugin RC: semver versioning (v1.14.3-rc.2 -> v1.14.4 now proposed).
- flux docker duplicates disabled (the "Cannot find replaceString" line). Two false comments
corrected.
Local dry run: regex deps 7 -> 20, branches 33 -> 29. The validator passes on 44.167.0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Run 350 cleared platform init (/user 200, /version 200, /repos 200), then stopped at
GET /api/v1/orgs/Home-Network with Gitea's own 403:
token does not have at least one of required scope(s), required=[read:organization],
token scope=read:package,write:repository,read:user
The workflow's header listed only those three scopes, and the operator minted exactly that.
Renovate's initRepo -> isOrg needs read:organization, and Gitea files /issues and /labels under
the issue scope, which the dashboard and PR labels need: write:issue.
- renovate.yaml: corrects the scope list (read:user, read:organization, write:repository,
write:issue, read:package), with the call each serves. A new preflight step probes them with
GETs, the token passed to curl on stdin, and names a missing scope instead of Renovate's bare
"Authentication failure".
- ci/lint.sh: rejects workflow `uses:` lines that need a Docker daemon (docker://,
renovatebot/github-action, docker/*-action), the class fixed in #84.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:
failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory
The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.
Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.
- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
:1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
`docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.
- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
(DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Also states in bind_browser_flow()'s docstring that it is reached only when git's
bind-browser-flow is "true".
Co-Authored-By: Claude Opus 5.5 <[email protected]>
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review findings on the passkey reconciler, fixed:
1. HIGH - the bind was automatic and rollback did not stick. A git-declared switch,
`bind-browser-flow` in passkey-desired-state.yaml, now decides whether pass 3 may
PUT browserFlow. Default "false": build + verify the candidate (or reuse a verified
one), log WOULD BIND <alias>, never touch the binding - including after an earlier
bind, so an operator's rollback sticks, and a hand-edited bound candidate is left
alone rather than shadowed by an -r2 build. "true" binds as before. The renderer
carries the one hand-edited line through and --check refuses anything but the
quoted strings "true"/"false". --no-bind can only narrow it.
2. HIGH - the protected-subtree guard read `alias` on sub-flow rows. Keycloak 26.0.6
AuthenticationManagementResource.recurseExecutions sets displayName =
flowRef.getAlias() and flowId on a sub-flow row and never sets alias (alias is the
CONFIG alias on authenticator rows), so the protected set was only the top-level
alias. Sub-flows are now named by displayName (getFlows() lists top-level flows
only, so flowId cannot be resolved through it); a sub-flow row without one leaves
the subtree undetermined and the flow pass fail-closed. The selftest stub now
returns 26.0.6-shaped rows, and a real run is asserted to refuse writes into
browser-mfa-forms and browser-mfa-conditional-otp.
3. MEDIUM - PUT /flows/{alias}/executions was screened on the path alias only, but
updateExecutions resolves the execution realm-wide by body id. body.id is now
screened against the protected ids AND required to be in the tree this run is
building (recorded from the build's own readback); the config POST is held to the
same build scope; and the body may carry only id/requirement/priority (the same
handler renames any flow named by flowId).
4. Stale "audit-only" docs rewritten (platform-config kustomization, chainsaw test,
desired-state header and its renderer template, README) to describe what ships.
5. backoffLimit 0 (a retried deterministic build failure burns the candidate-alias
budget); the stale "Renovate does not manage image: tags" comment corrected and the
image pinned by index digest.
6. The ~160-line header duplicated in the YAML and the script is now kept once, in the
script, with a short pointer above it.
Selftest: 259 assertions (was 216). mutations.py: 40 injected regressions (was 30),
including one per finding above, all caught.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.
A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.
pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of
GET/PUT /admin/realms/homelab, readback-verified.
pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE
literally-named alias; enabled, defaultAction false so enrolment stays
opt-in.
pass 3 the browser flow -- BLUE/GREEN.
PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:
1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
and zero user-visible effect.
2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
conditional-OTP config CONTENTS -- before anything is bound, so the
regression that once 403'd every platform-admin is caught BEFORE it bites.
3. BIND with ONE atomic PUT setting browserFlow.
4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
flow survives as the rollback target.
The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.
Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.
Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.
Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.
Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.
Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.
03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local => NOT INSIDE (3h45m early)
The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.
Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.
Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.
STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator ran the Renovate workflow manually and it failed. The cause is not
the workflow: renovate.json was never a valid Renovate config. It carried five
pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment,
_ignorePaths_comment, _docker_comment) because JSON has no comments, and
Renovate REJECTS unknown top-level options. Proven with the official validator:
ERROR: Found errors in configuration
Invalid configuration option: _comment
Invalid configuration option: _docker_comment
Invalid configuration option: _ignorePaths_comment
Invalid configuration option: _kubernetes_comment
Invalid configuration option: _unmanageable_comment
This is the SECOND independent reason Renovate never worked, on top of PR #77's
cron firing outside renovate.json's own schedule window. Either alone was enough
to produce "nothing ever happens"; #73 shipped both.
Converted renovate.json -> renovate.json5. JSON5 permits real // comments and
Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the
old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original
rationale is preserved as positioned comments (74 comment lines) rather than
flattened into a description string.
Verified the conversion dropped nothing: 18 non-comment keys before and after,
no missing/extra keys, no changed values, docker: {enabled: true} still present.
The validator now returns "Config validated successfully".
DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of
needing to be `custom.regex` (which would silently disable every inline
`# renovate:` marker). The validator accepts `regex`; unfounded, unchanged.
Renovate has still never completed a run -- this fixes one of two blockers and
nothing in the gate catches an invalid config. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:
pihole 192.168.2.23 -> 192.168.2.20 TTL=0
public 1.1.1.1 -> wan.home.keithjackson.dev TTL=300
router 192.168.2.1 -> wan.... TTL=153 -> 148 -> 143 (decrementing = caching)
dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.
So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".
Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.
apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.
UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ha-control-plane-as-code/check.sh had:
re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:...'secondary'",
K3S_TEXT, re.S)
With re.S every `.` matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?`
can each consume the whole document -- nested unbounded repetition, which
backtracks exponentially on a failed or late match. MEASURED in CI via
`docker top` on the live job container: python3 at 19:54 of CPU time, which is
essentially the entire ~28 minute gate.
Localised to the exact line by extracting the inline python and running it under
faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from
this call -- not inferred from reading.
Replaced `.` with `[^\n]*` and dropped re.S, so the {0,10} is a real ten-line
window with nothing to backtrack into. On the real input (22,179 chars):
0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical
success message.
WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every *.yml in
roles/k3s_server concatenated. The regex was latent for as long as it existed;
PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing
the input past the point where exponential backtracking stopped being
survivable.
WHY ONLY infra/** WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so
the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which
selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and
#72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran
~28m.
The pattern is unique in the repo (grep over all l0-static check.sh files).
Not re-timed in CI post-fix; this PR's own run is the first datapoint.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The first real run failed on all three nodes:
fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"
update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.
Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.
So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
running kernel is not the newest installed, using `linux-version` (from
linux-base, a dependency of every linux-image, measured present) because
kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.
The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.
Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.
Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster
DNS was a single pod on a single node, so draining or rebooting that node was a
CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service
lookup, every CNPG replica finding its primary, every operator addressing the
API by name. It was invisible because 1/1 Ready looks healthy.
It is urgent rather than latent now: the scheduled reboot automation on
feat/automated-upgrades drains and reboots one node at a time, unattended,
overnight -- turning a single point of failure into a recurring one.
WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns
Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml,
managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash
label. k3s re-stages its bundled manifests on startup, so replicas: 2 written
there -- or a kubectl scale -- is reverted on the next restart or upgrade.
Rejected alternatives:
* coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern).
Deterministic, but takes ownership of coredns from k3s permanently so k3s
upgrades stop updating it. Too high a price for a replica count.
* A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy
controller would both own one object and fight -- the default-deny
NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit.
An HPA is a separate object k3s does not own, driving /scale rather than the
Deployment spec, so there is no shared-ownership fight; if k3s re-stages
replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief
window at 1 replica after a k3s restart.
It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the
autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%),
so the 80% target is never approached. scaleDown stabilization 600s because DNS
is the worst thing to flap; scaleUp immediate, since that restores redundancy.
The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds
the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched.
UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on
restart is reasoned, not observed. After merge check `describe hpa coredns` for
ScalingActive=True -- an inert HPA looks identical to a working one because the
hand scale already left 2 replicas. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two halves of one repeatable upgrade process. Neither works alone.
(a) OS upgrades and reboots
platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.
infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.
The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.
Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.
Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).
alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.
Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.
Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.
platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.
(b) Image/dependency automation
Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.
.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.
Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.
conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.
Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.
NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.
Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).
Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.
No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.
UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC
discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is
not serving yet, the shared Gateway answers 503, and the proxy dies:
[provider.go:55] Performing OIDC Discovery...
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover
OIDC configuration: unexpected status "503": no healthy upstream
It then sits in exponential CrashLoopBackOff long after Keycloak is healthy --
one replica was still down minutes later and only a manual pod delete cleared
it. There is no retry flag, a readinessProbe cannot help (the container is
already dead), and CrashLoopBackOff backoff is not tunable per-pod.
Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc
until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries
promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the
worst case is exactly today's behaviour and never something new.
extraInitContainers is the chart's OWN hook for this -- its commented example is
literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned
chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values
key is silently ignored. No new base image: the chart already runs an alpine init
container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream)
rather than the chart's `latest`.
Rejected: Flux postRenderers (no precedent in this tree, and the chart has a
hook); --skip-oidc-discovery (removes the fetch entirely, but needs four
hardcoded endpoints and I could not verify lazy-JWKS offline -- this component
gates SSO for seven apps, so the additive option wins).
Shell vars are written BARE with no braced form: platform declares
postBuild.substituteFrom, and envsubst would blank a braced shell variable and
silently break the loop. Same convention as the gitea/nextcloud OIDC hooks.
UNAPPLIED; the init container has never run, and the race was not re-created to
prove the fix. Check the init container's log on first apply. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
All three nodes had to be powered off to move them onto a UPS, and there was
NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the
decommissioned dev cluster. The sequence was improvised live. Two defects came
out of doing it.
1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No
shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s,
containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG
instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes.
roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and
shutdown-grace-period-critical-pods=20s.
The half that silently defeats the other half: this is enforced by LOGIND,
not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits
at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless,
truncating the drain with no error. The role ships a logind.conf.d drop-in
(120) and ASSERTS cap >= grace period, because the failure is invisible.
2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki
carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the
quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so
that PVC was the only copy; 72h retention bounded the loss, the mechanism
does not. Symptom was the volume count going 25 -> 24, first misread as a
miscount. Repaired: scale back to the declared 1 recreated the PVC and
loki-0 is 2/2 Running.
New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence
that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays
running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats
a deletion. Discovery is computed from the live cluster, so a newly onboarded
service needs no edit here.
runbooks/cluster-maintenance-shutdown.md is the operator page: order in both
directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the
un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is
fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding
volume and the self-recovering keycloak-operator from the one thing that needs
a human (a stuck oauth2-proxy replica).
THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s
restart, which is an operator action. Review config.yaml's kubelet-arg before
applying: a bad config.yaml is how k3s fails to come up. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85
