The 26.x Keycloak operator creates its own ServiceMonitor named after the
CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled
defaults true. That collides with the Flux-owned platform/keycloak/
servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR
stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls
every downstream Kustomization.
Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are
written against it; the operator's scrapes the same port 9000 /metrics) and
set spec.serviceMonitor.enabled: false.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Replace the deprecated top-level `docker` key with the packageRule Renovate's
own migration produced (matchCategories docker, enabled true), keeping every
comment. Supersedes #139, which dropped all comment lines.
- Add gitea/runner (act_runner's upstream rename) to the blast-radius rule.
- rangeStrategy bump for flux/helm `>=A <B` ranges: the default `auto` resolves
to `replace`, which turned `>=1.12.0 <1.13.0` into `<1.14.0` (#131, #130,
#115). In-range `bump` updates are disabled; Flux adopts those unaided.
- Restore headlamp's lower bound: `<0.47.0` -> `>=0.46.0 <0.47.0` (live 0.46.0).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2
ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new
operator fails to start its keycloaksamlclients informer (Not Found) and
CrashLoops, holding Flux `platform` in health checks.
- add both CRD URLs with the same renovate marker/depName
- patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was
OOMKilled at 421/450Mi, 154 restarts)
- README/renovate comments: Renovate cannot add CRDs a release introduces
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The velero CLI and server are version-coupled, so VELERO_VERSION follows the
server running on prod (docker.io/velero/velero:v1.18.2, chart 12.2.3,
verified 2026-11-07). It has no renovate marker on purpose: bump it together
with the chart's appVersion. The verify stage asserts that
`velero version --client-only` reports the pin. The build arg reaches CI
through build-image.sh --print-build-args, so no .gitea/ or ci/ change is
needed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Gitea dropped the "1." prefix; 28.x is the successor of 1.27.
- image 28.3.0, update.range >=28.0.0 <29.0.0, Renovate ceiling <29.0.0
- drop GITEA__server__DOMAIN (no longer read in 28, ROOT_URL is the source)
- declare RUN_RETENTION_DAYS=400, EGRESS_MODE=lax (no allow list: the OIDC
login source is not gated by the egress policy in 28.3.0), and
DISABLE_REGISTRATION=true (28's new default)
- document autoRollbackOnFailedHealth as inert (no ImagePolicy is rendered);
rollback is a PVC restore because the DB migration is one-way
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add assertion (8) to internal-services-publish-no-public-dns. The external-dns
HelmRelease must declare annotationPrefix explicitly, and every
external-dns annotation in apps/, platform/, runbooks/ and catalog/render.py
must use that prefix. The v0.23.0 default-prefix change on 2026-11-07
ignored the controller: none, target and hostname annotations while
assertions (1)-(3) stayed green.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
external-dns v0.22 changed its default annotation prefix to
external-dns.kubernetes.io/ with no fallback. Renovate #129 moved the chart
to 1.23.0 (app v0.23.0), and at 12:13Z it ignored every alpha-prefixed
annotation in this repo: the Gateway target (public names went to A
192.168.2.20), controller: none (jackett, pihole and semaphore got
published), and the L4 Service hostnames (unifi and crafty-mc were deleted).
Set annotationPrefix: external-dns.alpha.kubernetes.io/ to restore the v0.21
reading. Floor the chart at 1.23.0, because the flag needs app >= 0.22.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Dispatcher image and PM_WORKER_IMAGE both move to
sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the
registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
`claude --print --output-format json` reports its own failures (API error, usage limit,
error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s
stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31.
describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails,
redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged)
makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry,
which admission requires, and the dispatcher's dry-run triage of #97 was read and approved.
PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the
configmap comments, the README and the plan's follow-up 3 is updated to match.
Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding
present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both
egress CNPs Valid.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule
refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry),
and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read
from the published layer).
Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's
dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The
renovate-digest markers are unchanged, so later bumps remain Renovate's.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
pihole/pihole (no tag) -> 2026.10.0 sha256:d3413471...
lscr.io/linuxserver/jackett:latest -> v0.24.2865-ls7 sha256:4fcf3cc2...
plexinc/pms-docker:latest -> 1.43.4.10903-e5521bd8c sha256:e0ab2739...
Each tag was resolved against its registry and matched the pods' imageID, so the image is
unchanged (the pod template string does change, so Flux rolls the three workloads).
renovate.json5: regex versioning for jackett (-lsN) and plex (build-hex); jackett is looked
up on ghcr.io via overridePackageName, because lscr.io is capped at dockerMaxPages=20 pages
and jackett's 32k tags put the running one at #31,980 (dry run proposed nothing without it);
jackett weekly (13 builds in 12 days); pihole deliberately unheld, year rollover = major.
Stale "publishes only latest" prose fixed in renovate.json5 and three runbooks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator decision. Both are held at every update type and labelled needs-review:
- quay.io/oauth2-proxy/oauth2-proxy (render.py constant, rendered sidecars, crafty). Since v7.13.0
oauth2-proxy validates refreshed OIDC sessions against the access token, every sidecar uses
--cookie-refresh, and that path is untested against this Keycloak.
- ghcr.io/renovatebot/renovate (and renovatebot/github-action if ever reintroduced). Nothing
re-validates renovate.json5 against the new version before it takes effect.
The validator passes on 44.167.0. Evaluated with Renovate's own packageRules engine: exactly 2 of 29
branch rows change (automerge true -> false, +needs-review); the other 27 are identical.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Audit of the first completed dry runs (353/354) plus a debug-level local reproduction:
- App-image updates edited only the generated apps/wave-*. A ServiceDefinition has no apiVersion,
so the kubernetes manager extracts nothing from catalog/, and every such PR would fail
render-golden (reproduced: GOLDEN DRIFT). New regex managers for catalog source.image and
render.py OAUTH2_PROXY_IMAGE put the source and generated edits in one branch (render-golden
rc 0, reproduced).
- CNPG Cluster imageName was untracked (3 Clusters), while the grant Job was tracked. New manager.
- allowedVersions mirror catalog update.range: this drops gitea 28.x, qbittorrent 20.x, radarr 6,
nextcloud 35 and postgresql 18.
- Never auto-merge: gitea, act_runner, postgresql (blast radius), flux range widening, python
minors. Majors wait for a Dependency Dashboard tick.
- velero plugin RC: semver versioning (v1.14.3-rc.2 -> v1.14.4 now proposed).
- flux docker duplicates disabled (the "Cannot find replaceString" line). Two false comments
corrected.
Local dry run: regex deps 7 -> 20, branches 33 -> 29. The validator passes on 44.167.0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Run 350 cleared platform init (/user 200, /version 200, /repos 200), then stopped at
GET /api/v1/orgs/Home-Network with Gitea's own 403:
token does not have at least one of required scope(s), required=[read:organization],
token scope=read:package,write:repository,read:user
The workflow's header listed only those three scopes, and the operator minted exactly that.
Renovate's initRepo -> isOrg needs read:organization, and Gitea files /issues and /labels under
the issue scope, which the dashboard and PR labels need: write:issue.
- renovate.yaml: corrects the scope list (read:user, read:organization, write:repository,
write:issue, read:package), with the call each serves. A new preflight step probes them with
GETs, the token passed to curl on stdin, and names a missing scope instead of Renovate's bare
"Authentication failure".
- ci/lint.sh: rejects workflow `uses:` lines that need a Docker daemon (docker://,
renovatebot/github-action, docker/*-action), the class fixed in #84.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:
failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory
The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.
Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git
blockers to going live go away:
- deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3...,
tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged.
- PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not
just cautious: the run-340 image's baked worker template predates imagePullSecrets, so
admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps.
- Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under
read:user. The documented list was missing it. Now read:user, write:repository,
write:issue, read:package everywhere.
- Org-package read documented as a team (Packages=Read, no repos), not just a scope.
- sops is toolbox-only; the .examples say so and give the in-place edit.
- runbook: key-name check no longer emits Secret values before the pipe.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.
pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.
ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.
The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.
kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.
The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.
- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
:1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
`docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.
- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
(DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Also states in bind_browser_flow()'s docstring that it is reached only when git's
bind-browser-flow is "true".
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.
A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.
pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of
GET/PUT /admin/realms/homelab, readback-verified.
pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE
literally-named alias; enabled, defaultAction false so enrolment stays
opt-in.
pass 3 the browser flow -- BLUE/GREEN.
PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:
1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
and zero user-visible effect.
2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
conditional-OTP config CONTENTS -- before anything is bound, so the
regression that once 403'd every platform-admin is caught BEFORE it bites.
3. BIND with ONE atomic PUT setting browserFlow.
4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
flow survives as the rollback target.
The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.
Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.
Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.
Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.
Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.
Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.
03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local => NOT INSIDE (3h45m early)
The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.
Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.
Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.
STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator ran the Renovate workflow manually and it failed. The cause is not
the workflow: renovate.json was never a valid Renovate config. It carried five
pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment,
_ignorePaths_comment, _docker_comment) because JSON has no comments, and
Renovate REJECTS unknown top-level options. Proven with the official validator:
ERROR: Found errors in configuration
Invalid configuration option: _comment
Invalid configuration option: _docker_comment
Invalid configuration option: _ignorePaths_comment
Invalid configuration option: _kubernetes_comment
Invalid configuration option: _unmanageable_comment
This is the SECOND independent reason Renovate never worked, on top of PR #77's
cron firing outside renovate.json's own schedule window. Either alone was enough
to produce "nothing ever happens"; #73 shipped both.
Converted renovate.json -> renovate.json5. JSON5 permits real // comments and
Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the
old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original
rationale is preserved as positioned comments (74 comment lines) rather than
flattened into a description string.
Verified the conversion dropped nothing: 18 non-comment keys before and after,
no missing/extra keys, no changed values, docker: {enabled: true} still present.
The validator now returns "Config validated successfully".
DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of
needing to be `custom.regex` (which would silently disable every inline
`# renovate:` marker). The validator accepts `regex`; unfounded, unchanged.
Renovate has still never completed a run -- this fixes one of two blockers and
nothing in the gate catches an invalid config. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:
pihole 192.168.2.23 -> 192.168.2.20 TTL=0
public 1.1.1.1 -> wan.home.keithjackson.dev TTL=300
router 192.168.2.1 -> wan.... TTL=153 -> 148 -> 143 (decrementing = caching)
dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.
So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".
Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.
apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.
UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ha-control-plane-as-code/check.sh had:
re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:...'secondary'",
K3S_TEXT, re.S)
With re.S every `.` matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?`
can each consume the whole document -- nested unbounded repetition, which
backtracks exponentially on a failed or late match. MEASURED in CI via
`docker top` on the live job container: python3 at 19:54 of CPU time, which is
essentially the entire ~28 minute gate.
Localised to the exact line by extracting the inline python and running it under
faulthandler.dump_traceback_later, which named re/__init__.py:177 in search from
this call -- not inferred from reading.
Replaced `.` with `[^\n]*` and dropped re.S, so the {0,10} is a real ten-line
window with nothing to backtrack into. On the real input (22,179 chars):
0.2 ms, match still found. Whole check: 0.199s, rc 0, 115 assertions, identical
success message.
WHY IT SURFACED NOW, AND IT WAS MINE: K3S_TEXT is every *.yml in
roles/k3s_server concatenated. The regex was latent for as long as it existed;
PR #71 added ~100 lines to that role (kubelet graceful-shutdown config), growing
the input past the point where exponential backtracking stopped being
survivable.
WHY ONLY infra/** WAS SLOW: touched-requirements.sh maps infra/* to R-ENV-01, so
the L0 run becomes --include-scope whole-repo --requirements R-ENV-01, which
selects this whole-repo check. Doc/platform changes never did -- #69 1m55s and
#72 1m53s stayed fast while #71 27m44s, the #73 merge 27m46s and #75 all ran
~28m.
The pattern is unique in the repo (grep over all l0-static check.sh files).
Not re-timed in CI post-fix; this PR's own run is the first datapoint.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The first real run failed on all three nodes:
fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"
update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.
Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.
So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
running kernel is not the newest installed, using `linux-version` (from
linux-base, a dependency of every linux-image, measured present) because
kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.
The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.
Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.
Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster
DNS was a single pod on a single node, so draining or rebooting that node was a
CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service
lookup, every CNPG replica finding its primary, every operator addressing the
API by name. It was invisible because 1/1 Ready looks healthy.
It is urgent rather than latent now: the scheduled reboot automation on
feat/automated-upgrades drains and reboots one node at a time, unattended,
overnight -- turning a single point of failure into a recurring one.
WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns
Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml,
managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash
label. k3s re-stages its bundled manifests on startup, so replicas: 2 written
there -- or a kubectl scale -- is reverted on the next restart or upgrade.
Rejected alternatives:
* coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern).
Deterministic, but takes ownership of coredns from k3s permanently so k3s
upgrades stop updating it. Too high a price for a replica count.
* A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy
controller would both own one object and fight -- the default-deny
NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit.
An HPA is a separate object k3s does not own, driving /scale rather than the
Deployment spec, so there is no shared-ownership fight; if k3s re-stages
replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief
window at 1 replica after a k3s restart.
It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the
autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%),
so the 80% target is never approached. scaleDown stabilization 600s because DNS
is the worst thing to flap; scaleUp immediate, since that restores redundancy.
The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds
the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched.
UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on
restart is reasoned, not observed. After merge check `describe hpa coredns` for
ScalingActive=True -- an inert HPA looks identical to a working one because the
hand scale already left 2 replicas. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two halves of one repeatable upgrade process. Neither works alone.
(a) OS upgrades and reboots
platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.
infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.
The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.
Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.
Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).
alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.
Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.
Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.
platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.
(b) Image/dependency automation
Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.
.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.
Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.
conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.
Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.
NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.
Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).
Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.
No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.
UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5.5