Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally,
L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and
Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns
gains assertion (9) tying the Gateway target to the tunnel UUID, requiring
cloudflare-proxied=false on every service-source publisher, and keeping the
tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture
opts out of proxying.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's
credentials.json down root:root 0440 — unreadable to the container. cloudflared
logged "permission denied" and then the red-herring "Error locating origin cert".
- deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440)
- L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the
container's run-as uid/groups; mutation self-tested
- runbook: permission-denied troubleshooting entry, credential rotation section
- decisions/2026-11-08-cloudflared-creds-permission.md
Co-Authored-By: Claude Opus 5.5 <[email protected]>
servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo
Service, and no-plaintext-secrets requires a template's secret value to be a
bare placeholder.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Adds platform/cloudflared: a locally-managed tunnel whose ingress rules live in
git. LAN/VPN-only hosts (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) map to 404. The apex
and *.home go to cilium-gateway-shared.gateway over TLS with matchSNItoHost.
Last comes a catch-all 404. Nothing routes to the Kubernetes API.
- 2-replica Deployment, hard node spread, non-root, pinned digest, PDB
- default-deny plus an egress CNP: Cloudflare edge 7844/443, the Gateway, and
Gateway backends by gateway-access=shared minus the LAN-only namespaces
(needed because of the CVE-2025-30162 L7LB egress re-check)
- PodMonitor plus two alerts that fire on absent series, with promtool tests
- its own Flux Kustomization (dependsOn platform-config, wait:false, nothing
depends on it), plus SOPS decryption patches in both overlays
- placeholder credentials (SOPS-encrypted) so it can merge before enrolment
- L0 cloudflared-ingress-rules, and a cloudflared_ scrape owner
- runbooks/cloudflare-tunnel.md and decisions/2026-11-08-cloudflare-tunnel.md
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The 26.x Keycloak operator creates its own ServiceMonitor named after the
CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled
defaults true. That collides with the Flux-owned platform/keycloak/
servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR
stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls
every downstream Kustomization.
Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are
written against it; the operator's scrapes the same port 9000 /metrics) and
set spec.serviceMonitor.enabled: false.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Run 500 (this PR's first CI run) failed L0 prod-baremetal-not-dev-surface: act runs steps with
umask 0000, the checkout was world-writable, and Ansible (now present, via the toolbox) refuses
a world-writable directory as an ansible.cfg source. Reproduced and fixed locally in the pinned
image.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- conformance/lib/run_l1_offline.py: checks run in a thread pool (--jobs / CONF_OFFLINE_JOBS,
default min(8, usable CPUs)); results consumed in submission order, so log lines, JUnit,
step summary and the 0/1/3 aggregation are unchanged. Exports a smaller MUTATION_JOBS to
the children when parallel, so nested pools do not oversubscribe.
- keycloak-{passkey,client-secret}-reconciler/mutations.py: anchors checked first (serial),
then the --selftest subprocesses run in a pool (MUTATION_JOBS, default min(8, CPUs));
verdicts reported in MUTATIONS order. Output byte-identical to the serial loop.
- ci/lint.sh: yamllint over LINT_JOBS contiguous slices concurrently, per-slice output
buffered and printed in order; stage rc = highest slice rc.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The gate job now runs in gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e...
(the image images.yaml publishes from infra/devcontainer/Containerfile) instead of
catthehacker/ubuntu:act-22.04 plus a per-run download of yamllint, check-jsonschema, kyverno,
promtool and chainsaw. Local (dev.sh) and CI now gate on the identical toolchain.
- actions/checkout (Node) replaced by a plain-git step with fetch-depth-0 semantics; the
toolbox has no Node. The job token is not persisted into .git/config.
- The install step is replaced by a check that every ci/versions.env pin equals the image's
/etc/dev-toolchain-versions, so the pins stay enforced.
- Renovate follows the digest through a `# renovate-digest:` marker; github-actions manager
disabled for own images.
- install-tools.sh kept (merge-gate.yaml blueprint + alert-rules-unit-tested check use it).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The in-cluster runner ran at the default capacity 1 with no config file, so
gate / images / renovate runs on different refs queued behind each other
(29-759s waits; run 472 waited 759s behind a renovate burst).
- ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via
CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault()
applies the same defaults to a partial file as to no file; the
generate-config example is deliberately not used (its labels would override
the registered ones, force_pull differs from the default).
- Pod-template config-revision annotation so ConfigMap-only edits can roll it.
- dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi.
Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs.
- runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1).
apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the
renderer only places it in a wave), so it is the source, not generated output.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Replace the deprecated top-level `docker` key with the packageRule Renovate's
own migration produced (matchCategories docker, enabled true), keeping every
comment. Supersedes #139, which dropped all comment lines.
- Add gitea/runner (act_runner's upstream rename) to the blast-radius rule.
- rangeStrategy bump for flux/helm `>=A <B` ranges: the default `auto` resolves
to `replace`, which turned `>=1.12.0 <1.13.0` into `<1.14.0` (#131, #130,
#115). In-range `bump` updates are disabled; Flux adopts those unaided.
- Restore headlamp's lower bound: `<0.47.0` -> `>=0.46.0 <0.47.0` (live 0.46.0).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2
ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new
operator fails to start its keycloaksamlclients informer (Not Found) and
CrashLoops, holding Flux `platform` in health checks.
- add both CRD URLs with the same renovate marker/depName
- patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was
OOMKilled at 421/450Mi, 154 restarts)
- README/renovate comments: Renovate cannot add CRDs a release introduces
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #136 bumps the renderer's OAUTH2_PROXY_IMAGE; the container-workload
and egress-class (vpn + internet) goldens still pinned v7.8.1. Regenerated
with the check scripts' own invocation; each diff is the image tag only.
Also updates the crafty README's current-pin row.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The velero CLI and server are version-coupled, so VELERO_VERSION follows the
server running on prod (docker.io/velero/velero:v1.18.2, chart 12.2.3,
verified 2026-11-07). It has no renovate marker on purpose: bump it together
with the chart's appVersion. The verify stage asserts that
`velero version --client-only` reports the pin. The build arg reaches CI
through build-image.sh --print-build-args, so no .gitea/ or ci/ change is
needed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Gitea dropped the "1." prefix; 28.x is the successor of 1.27.
- image 28.3.0, update.range >=28.0.0 <29.0.0, Renovate ceiling <29.0.0
- drop GITEA__server__DOMAIN (no longer read in 28, ROOT_URL is the source)
- declare RUN_RETENTION_DAYS=400, EGRESS_MODE=lax (no allow list: the OIDC
login source is not gated by the egress policy in 28.3.0), and
DISABLE_REGISTRATION=true (28's new default)
- document autoRollbackOnFailedHealth as inert (no ImagePolicy is rendered);
rollback is a PVC restore because the DB migration is one-way
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add assertion (8) to internal-services-publish-no-public-dns. The external-dns
HelmRelease must declare annotationPrefix explicitly, and every
external-dns annotation in apps/, platform/, runbooks/ and catalog/render.py
must use that prefix. The v0.23.0 default-prefix change on 2026-11-07
ignored the controller: none, target and hostname annotations while
assertions (1)-(3) stayed green.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
external-dns v0.22 changed its default annotation prefix to
external-dns.kubernetes.io/ with no fallback. Renovate #129 moved the chart
to 1.23.0 (app v0.23.0), and at 12:13Z it ignored every alpha-prefixed
annotation in this repo: the Gateway target (public names went to A
192.168.2.20), controller: none (jackett, pihole and semaphore got
published), and the L4 Service hostnames (unifi and crafty-mc were deleted).
Set annotationPrefix: external-dns.alpha.kubernetes.io/ to restore the v0.21
reading. Floor the chart at 1.23.0, because the flag needs app >= 0.22.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Dispatcher image and PM_WORKER_IMAGE both move to
sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the
registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
`claude --print --output-format json` reports its own failures (API error, usage limit,
error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s
stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31.
describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails,
redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged)
makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry,
which admission requires, and the dispatcher's dry-run triage of #97 was read and approved.
PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the
configmap comments, the README and the plan's follow-up 3 is updated to match.
Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding
present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both
egress CNPs Valid.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule
refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry),
and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read
from the published layer).
Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's
dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The
renovate-digest markers are unchanged, so later bumps remain Renovate's.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
pihole/pihole (no tag) -> 2026.10.0 sha256:d3413471...
lscr.io/linuxserver/jackett:latest -> v0.24.2865-ls7 sha256:4fcf3cc2...
plexinc/pms-docker:latest -> 1.43.4.10903-e5521bd8c sha256:e0ab2739...
Each tag was resolved against its registry and matched the pods' imageID, so the image is
unchanged (the pod template string does change, so Flux rolls the three workloads).
renovate.json5: regex versioning for jackett (-lsN) and plex (build-hex); jackett is looked
up on ghcr.io via overridePackageName, because lscr.io is capped at dockerMaxPages=20 pages
and jackett's 32k tags put the running one at #31,980 (dry run proposed nothing without it);
jackett weekly (13 builds in 12 days); pihole deliberately unheld, year rollover = major.
Stale "publishes only latest" prose fixed in renovate.json5 and three runbooks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Keycloak CR has no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the
server image; the three keycloak-k8s-resources release URLs in
platform/keycloak/kustomization.yaml are therefore THE Keycloak version, and Renovate
could not see them (no `# renovate:` marker). They now carry markers under one depName
(one branch, groupName keycloak-operator), with a packageRule: automerge false, labels
needs-review + blast-radius; majors still wait on the dashboard.
The untagged quay.io/keycloak/keycloak in catalog/keycloak.yaml renders
apps/wave-1/keycloak.yaml, which no kustomization lists. It is pinned to the operator's
26.0.6 as documentation, re-rendered, and Renovate is disabled for that image so it
cannot raise PRs against a file nothing deploys. The render is kept (not deleted): it is
the fixture for apps/wave-1/tests/app-slis.test.yaml's keycloak SLI rule tests.
README: an "Upgrading Keycloak" section — an operator bump must be checked against
passkey-reconciler.yaml's 26.0.6 Admin API assumptions.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator decision. Both are held at every update type and labelled needs-review:
- quay.io/oauth2-proxy/oauth2-proxy (render.py constant, rendered sidecars, crafty). Since v7.13.0
oauth2-proxy validates refreshed OIDC sessions against the access token, every sidecar uses
--cookie-refresh, and that path is untested against this Keycloak.
- ghcr.io/renovatebot/renovate (and renovatebot/github-action if ever reintroduced). Nothing
re-validates renovate.json5 against the new version before it takes effect.
The validator passes on 44.167.0. Evaluated with Renovate's own packageRules engine: exactly 2 of 29
branch rows change (automerge true -> false, +needs-review); the other 27 are identical.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Audit of the first completed dry runs (353/354) plus a debug-level local reproduction:
- App-image updates edited only the generated apps/wave-*. A ServiceDefinition has no apiVersion,
so the kubernetes manager extracts nothing from catalog/, and every such PR would fail
render-golden (reproduced: GOLDEN DRIFT). New regex managers for catalog source.image and
render.py OAUTH2_PROXY_IMAGE put the source and generated edits in one branch (render-golden
rc 0, reproduced).
- CNPG Cluster imageName was untracked (3 Clusters), while the grant Job was tracked. New manager.
- allowedVersions mirror catalog update.range: this drops gitea 28.x, qbittorrent 20.x, radarr 6,
nextcloud 35 and postgresql 18.
- Never auto-merge: gitea, act_runner, postgresql (blast radius), flux range widening, python
minors. Majors wait for a Dependency Dashboard tick.
- velero plugin RC: semver versioning (v1.14.3-rc.2 -> v1.14.4 now proposed).
- flux docker duplicates disabled (the "Cannot find replaceString" line). Two false comments
corrected.
Local dry run: regex deps 7 -> 20, branches 33 -> 29. The validator passes on 44.167.0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Run 350 cleared platform init (/user 200, /version 200, /repos 200), then stopped at
GET /api/v1/orgs/Home-Network with Gitea's own 403:
token does not have at least one of required scope(s), required=[read:organization],
token scope=read:package,write:repository,read:user
The workflow's header listed only those three scopes, and the operator minted exactly that.
Renovate's initRepo -> isOrg needs read:organization, and Gitea files /issues and /labels under
the issue scope, which the dashboard and PR labels need: write:issue.
- renovate.yaml: corrects the scope list (read:user, read:organization, write:repository,
write:issue, read:package), with the call each serves. A new preflight step probes them with
GETs, the token passed to curl on stdin, and names a missing scope instead of Renovate's bare
"Authentication failure".
- ci/lint.sh: rejects workflow `uses:` lines that need a Docker daemon (docker://,
renovatebot/github-action, docker/*-action), the class fixed in #84.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:
failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory
The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.
Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git
blockers to going live go away:
- deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3...,
tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged.
- PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not
just cautious: the run-340 image's baked worker template predates imagePullSecrets, so
admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps.
- Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under
read:user. The documented list was missing it. Now read:user, write:repository,
write:issue, read:package everywhere.
- Org-package read documented as a team (Packages=Read, no repos), not just a scope.
- sops is toolbox-only; the .examples say so and give the in-place edit.
- runbook: key-name check no longer emits Secret values before the pipe.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.
pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.
ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.
The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.
kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.
The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.
- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
:1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
`docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.
- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
(DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Also states in bind_browser_flow()'s docstring that it is reached only when git's
bind-browser-flow is "true".
Co-Authored-By: Claude Opus 5.5 <[email protected]>
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review findings on the passkey reconciler, fixed:
1. HIGH - the bind was automatic and rollback did not stick. A git-declared switch,
`bind-browser-flow` in passkey-desired-state.yaml, now decides whether pass 3 may
PUT browserFlow. Default "false": build + verify the candidate (or reuse a verified
one), log WOULD BIND <alias>, never touch the binding - including after an earlier
bind, so an operator's rollback sticks, and a hand-edited bound candidate is left
alone rather than shadowed by an -r2 build. "true" binds as before. The renderer
carries the one hand-edited line through and --check refuses anything but the
quoted strings "true"/"false". --no-bind can only narrow it.
2. HIGH - the protected-subtree guard read `alias` on sub-flow rows. Keycloak 26.0.6
AuthenticationManagementResource.recurseExecutions sets displayName =
flowRef.getAlias() and flowId on a sub-flow row and never sets alias (alias is the
CONFIG alias on authenticator rows), so the protected set was only the top-level
alias. Sub-flows are now named by displayName (getFlows() lists top-level flows
only, so flowId cannot be resolved through it); a sub-flow row without one leaves
the subtree undetermined and the flow pass fail-closed. The selftest stub now
returns 26.0.6-shaped rows, and a real run is asserted to refuse writes into
browser-mfa-forms and browser-mfa-conditional-otp.
3. MEDIUM - PUT /flows/{alias}/executions was screened on the path alias only, but
updateExecutions resolves the execution realm-wide by body id. body.id is now
screened against the protected ids AND required to be in the tree this run is
building (recorded from the build's own readback); the config POST is held to the
same build scope; and the body may carry only id/requirement/priority (the same
handler renames any flow named by flowId).
4. Stale "audit-only" docs rewritten (platform-config kustomization, chainsaw test,
desired-state header and its renderer template, README) to describe what ships.
5. backoffLimit 0 (a retried deterministic build failure burns the candidate-alias
budget); the stale "Renovate does not manage image: tags" comment corrected and the
image pinned by index digest.
6. The ~160-line header duplicated in the YAML and the script is now kept once, in the
script, with a short pointer above it.
Selftest: 259 assertions (was 216). mutations.py: 40 injected regressions (was 30),
including one per finding above, all caught.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.
A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.
pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of
GET/PUT /admin/realms/homelab, readback-verified.
pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE
literally-named alias; enabled, defaultAction false so enrolment stays
opt-in.
pass 3 the browser flow -- BLUE/GREEN.
PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:
1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
and zero user-visible effect.
2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
conditional-OTP config CONTENTS -- before anything is bound, so the
regression that once 403'd every platform-admin is caught BEFORE it bites.
3. BIND with ONE atomic PUT setting browserFlow.
4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
flow survives as the rollback target.
The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.
Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.
Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.
Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.
Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.
Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.
03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local => NOT INSIDE (3h45m early)
The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.
Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.
Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.
STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator ran the Renovate workflow manually and it failed. The cause is not
the workflow: renovate.json was never a valid Renovate config. It carried five
pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment,
_ignorePaths_comment, _docker_comment) because JSON has no comments, and
Renovate REJECTS unknown top-level options. Proven with the official validator:
ERROR: Found errors in configuration
Invalid configuration option: _comment
Invalid configuration option: _docker_comment
Invalid configuration option: _ignorePaths_comment
Invalid configuration option: _kubernetes_comment
Invalid configuration option: _unmanageable_comment
This is the SECOND independent reason Renovate never worked, on top of PR #77's
cron firing outside renovate.json's own schedule window. Either alone was enough
to produce "nothing ever happens"; #73 shipped both.
Converted renovate.json -> renovate.json5. JSON5 permits real // comments and
Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the
old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original
rationale is preserved as positioned comments (74 comment lines) rather than
flattened into a description string.
Verified the conversion dropped nothing: 18 non-comment keys before and after,
no missing/extra keys, no changed values, docker: {enabled: true} still present.
The validator now returns "Config validated successfully".
DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of
needing to be `custom.regex` (which would silently disable every inline
`# renovate:` marker). The validator accepts `regex`; unfounded, unchanged.
Renovate has still never completed a run -- this fixes one of two blockers and
nothing in the gate catches an invalid config. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85
