Compare commits

..
Author SHA1 Message Date
jakman85 50c8719b12 Merge pull request 'fix(vpn): default client profiles to MTU 1280; correct the VPN runbook' (#159) from fix/wg-client-mtu-docs into main
gate / gate (push) Successful in 57s
gate / conformance (push) Skipped
Reviewed-on: #159
2026-11-09 03:26:02 +00:00
jakman85andClaude Opus 5.5 8f60e6103a fix(vpn): default client profiles to MTU 1280; correct the VPN runbook
gate / gate (pull_request) Successful in 57s
gate / conformance (pull_request) Skipped
new-peer.sh generated MTU 1420, which stalled traffic on mobile data
despite a healthy handshake (one phone, one carrier, 2026-11-09). Default
to 1280, overridable through WG_MTU.

Correct stale claims in the runbook and the DaemonSet header against
live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a
SNATed host identity. Add a troubleshooting section.

The DaemonSet render is unchanged, so WireGuard pods do not roll.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 21:15:20 -06:00
jakman85 da0515c8c8 Merge pull request 'fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS' (#158) from fix/wg-home-peer-pihole-sources into main
gate / gate (push) Successful in 55s
gate / conformance (push) Skipped
Reviewed-on: #158
2026-11-09 02:50:04 +00:00
jakman85 3a2ec5a12c Merge pull request 'feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard' (#157) from feat/pihole-explicit-lan-names into main
gate / gate (push) Canceled after 25s
gate / conformance (push) Canceled after 0s
Reviewed-on: #157
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 02:49:39 +00:00
jakman85andClaude Opus 5.5 f13d336630 fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS
gate / gate (pull_request) Successful in 1m12s
gate / conformance (pull_request) Skipped
Add the at-home peer (10.90.0.3) and list ${WG_CLIENT_CIDR} in the PiHole
DNS frontends' loadBalancerSourceRanges. wg0 is a Cilium device, so tunnel
clients reach the source-range check untranslated; a LAN-only list is
predicted to drop their DNS. Adds an L0 guard that both ranges are present
and every range is RFC1918, and corrects the README's SNAT and IPv6 claims.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 20:45:48 -06:00
jakman85andClaude Opus 5.5 8fe8d5df8a feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard
gate / gate (pull_request) Successful in 1m14s
gate / conformance (pull_request) Skipped
Public names are proxied CNAMEs to the Cloudflare Tunnel since PR #156 and
work from the LAN, so they should resolve publicly there too. Replace the
address=/${BASE_DOMAIN}/ wildcard with explicit entries: the LAN-only names
(derived from every controller:none HTTPRoute), gitea/nextcloud (off
Cloudflare's 100 MB / 100 s limits), keycloak/auth, and the L4 VIPs.

Add L0 check pihole-lan-names, mutation-tested. Correct docs that still
described public names as CNAME to wan. and the cloudflared entrypoint comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 19:51:49 -06:00
jakman85 020e736541 Merge pull request 'feat(dns): public names via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only' (#156) from feat/dns-via-cloudflare-tunnel into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #156
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 01:27:06 +00:00
jakman85andClaude Opus 5.5 92944486d1 feat(dns): publish public names as proxied CNAMEs to the Cloudflare Tunnel
gate / gate (pull_request) Successful in 45s
gate / conformance (pull_request) Skipped
Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally,
L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and
Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns
gains assertion (9) tying the Gateway target to the tunnel UUID, requiring
cloudflare-proxied=false on every service-source publisher, and keeping the
tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture
opts out of proxying.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 17:55:41 -06:00
jakman85 2e777e13ca Merge pull request 'chore(cloudflared): rotate tunnel credential (old secret exposed in a transcript)' (#155) from feature/cloudflare-tunnel-config into main
gate / gate (push) Successful in 41s
gate / conformance (push) Skipped
Reviewed-on: #155
2026-11-08 22:11:24 +00:00
jakman85 063c8acf77 chore(cloudflared): rotate tunnel credential (old secret exposed in a transcript)
gate / gate (pull_request) Successful in 41s
gate / conformance (pull_request) Skipped
2026-11-08 16:05:11 -06:00
jakman85 16222606ae Merge pull request 'fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret' (#154) from fix/cloudflared-creds-permission into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #154
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 21:57:46 +00:00
jakman85andClaude Opus 5.5 d9ab57f2d1 fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret
gate / gate (pull_request) Successful in 42s
gate / conformance (pull_request) Skipped
The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's
credentials.json down root:root 0440 — unreadable to the container. cloudflared
logged "permission denied" and then the red-herring "Error locating origin cert".

- deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440)
- L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the
  container's run-as uid/groups; mutation self-tested
- runbook: permission-denied troubleshooting entry, credential rotation section
- decisions/2026-11-08-cloudflared-creds-permission.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 15:49:56 -06:00
jakman85 9724445a08 Merge pull request 'Added cloudflare creds and tunnel config' (#153) from feature/cloudflare-tunnel-config into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #153
2026-11-08 21:34:24 +00:00
jakman85 32b7d5bac1 Added cloudflare creds and tunnel config
gate / gate (pull_request) Successful in 42s
gate / conformance (pull_request) Skipped
2026-11-08 15:30:19 -06:00
jakman85 83e0ac194f Merge pull request 'feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change)' (#152) from feat/cloudflare-tunnel into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #152
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 18:43:42 +00:00
jakman85andClaude Opus 5.5 09f219fe7f fix(cloudflared): auditable Service+ServiceMonitor scrape; placeholder-only template value
gate / gate (pull_request) Successful in 42s
gate / conformance (pull_request) Skipped
servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo
Service, and no-plaintext-secrets requires a template's secret value to be a
bare placeholder.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 12:39:56 -06:00
jakman85andClaude Opus 5.5 aabc8c60b6 feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change)
Adds platform/cloudflared: a locally-managed tunnel whose ingress rules live in
git. LAN/VPN-only hosts (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) map to 404. The apex
and *.home go to cilium-gateway-shared.gateway over TLS with matchSNItoHost.
Last comes a catch-all 404. Nothing routes to the Kubernetes API.

- 2-replica Deployment, hard node spread, non-root, pinned digest, PDB
- default-deny plus an egress CNP: Cloudflare edge 7844/443, the Gateway, and
  Gateway backends by gateway-access=shared minus the LAN-only namespaces
  (needed because of the CVE-2025-30162 L7LB egress re-check)
- PodMonitor plus two alerts that fire on absent series, with promtool tests
- its own Flux Kustomization (dependsOn platform-config, wait:false, nothing
  depends on it), plus SOPS decryption patches in both overlays
- placeholder credentials (SOPS-encrypted) so it can merge before enrolment
- L0 cloudflared-ingress-rules, and a cloudflared_ scrape owner
- runbooks/cloudflare-tunnel.md and decisions/2026-11-08-cloudflare-tunnel.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 12:38:22 -06:00
renovate-bot eb62c13c6b Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 63e1aa7' (#151) from renovate/gitea.home.keithjackson.dev-home-network-conformance-runner-main into main
gate / gate (push) Successful in 41s
gate / conformance (push) Skipped
2026-11-08 14:15:18 +00:00
renovate-bot 4226292dd6 Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to 89ce7ee' (#150) from renovate/gitea.home.keithjackson.dev-home-network-agent-runner-main into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
2026-11-08 08:17:22 +00:00
renovate-bot 213429a41f chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 63e1aa7
gate / gate (pull_request) Successful in 41s
gate / conformance (pull_request) Skipped
2026-11-08 08:16:28 +00:00
renovate-bot 0120bc10d4 chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to 89ce7ee
gate / gate (pull_request) Successful in 52s
gate / conformance (pull_request) Skipped
2026-11-08 08:16:27 +00:00
jakman85 d1963bd000 Merge pull request 'chore(deps): Update Helm release longhorn to <1.14.0' (#131) from renovate/longhorn-1.x into main
gate / gate (push) Failing after 40s
gate / conformance (push) Skipped
Reviewed-on: #131
2026-11-08 06:55:55 +00:00
jakman85 0b0b5abc8e Merge pull request 'feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools' (#149) from feat/parallel-l0-checks into main
gate / gate (push) Successful in 39s
gate / conformance (push) Skipped
Reviewed-on: #149
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:53:42 +00:00
jakman85 6e2e353bcc Merge pull request 'feat(ci): run the merge gate inside the CI-built toolbox image, pinned by digest' (#147) from feat/gate-toolbox-container into main
gate / gate (push) Successful in 2m33s
gate / conformance (push) Skipped
Reviewed-on: #147
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:51:00 +00:00
jakman85 045d81c7a2 Merge pull request 'feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs' (#146) from feat/runner-capacity-2 into main
gate / gate (push) Canceled after 1m15s
gate / conformance (push) Canceled after 0s
Reviewed-on: #146
2026-11-08 06:47:05 +00:00
jakman85 7be693bf37 Merge pull request 'chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6' (#136) from renovate/quay.io-oauth2-proxy-oauth2-proxy-7.x into main
gate / gate (push) Canceled after 1m32s
gate / conformance (push) Canceled after 0s
Reviewed-on: #136
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:45:31 +00:00
jakman85 9e9a3ea8cc Merge pull request 'fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)' (#148) from fix/keycloak-servicemonitor-owner into main
gate / gate (push) Successful in 3m0s
gate / conformance (push) Skipped
Reviewed-on: #148
2026-11-08 06:37:13 +00:00
jakman85andClaude Opus 5.5 89f48e4a5f docs(decisions): handoff for parallel L0 checks; runner usage names --jobs
gate / gate (pull_request) Successful in 59s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:34:48 -06:00
jakman85andClaude Opus 5.5 64c5617ddd docs(decisions): record CI runs 500/503 for gate-toolbox-container
gate / gate (pull_request) Successful in 2m20s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:26:27 -06:00
jakman85andClaude Opus 5.5 35ba875380 fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)
gate / gate (pull_request) Successful in 3m2s
gate / conformance (pull_request) Skipped
The 26.x Keycloak operator creates its own ServiceMonitor named after the
CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled
defaults true. That collides with the Flux-owned platform/keycloak/
servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR
stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls
every downstream Kustomization.

Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are
written against it; the operator's scrapes the same port 9000 /metrics) and
set spec.serviceMonitor.enabled: false.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:25:45 -06:00
jakman85andClaude Opus 5.5 27d603f726 fix(ci): umask 0022 in the git checkout step — act's 0000 made Ansible ignore ansible.cfg
gate / gate (pull_request) Failing after 2m20s
gate / conformance (pull_request) Skipped
Run 500 (this PR's first CI run) failed L0 prod-baremetal-not-dev-surface: act runs steps with
umask 0000, the checkout was world-writable, and Ansible (now present, via the toolbox) refuses
a world-writable directory as an ansible.cfg source. Reproduced and fixed locally in the pinned
image.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:20:38 -06:00
jakman85andClaude Opus 5.5 a775082ae1 feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools
- conformance/lib/run_l1_offline.py: checks run in a thread pool (--jobs / CONF_OFFLINE_JOBS,
  default min(8, usable CPUs)); results consumed in submission order, so log lines, JUnit,
  step summary and the 0/1/3 aggregation are unchanged. Exports a smaller MUTATION_JOBS to
  the children when parallel, so nested pools do not oversubscribe.
- keycloak-{passkey,client-secret}-reconciler/mutations.py: anchors checked first (serial),
  then the --selftest subprocesses run in a pool (MUTATION_JOBS, default min(8, CPUs));
  verdicts reported in MUTATIONS order. Output byte-identical to the serial loop.
- ci/lint.sh: yamllint over LINT_JOBS contiguous slices concurrently, per-slice output
  buffered and printed in order; stage rc = highest slice rc.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:18:57 -06:00
jakman85 1a33814522 Merge pull request 'chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds' (#145) from chore/renovate-config-cleanup into main
gate / gate (push) Successful in 2m59s
gate / conformance (push) Skipped
Reviewed-on: #145
2026-11-08 06:17:59 +00:00
jakman85 4a3f6dee59 Merge pull request 'fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)' (#144) from fix/keycloak-operator-crds into main
gate / gate (push) Canceled after 0s
gate / conformance (push) Canceled after 0s
Reviewed-on: #144
2026-11-08 06:16:56 +00:00
jakman85andClaude Opus 5.5 514ed5d5a4 docs(decisions): record local gate result for gate-toolbox-container
gate / gate (pull_request) Failing after 2m31s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:15:38 -06:00
jakman85andClaude Opus 5.5 9537ca2ddb feat(ci): run the merge gate inside the CI-built toolbox image, pinned by digest
The gate job now runs in gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e...
(the image images.yaml publishes from infra/devcontainer/Containerfile) instead of
catthehacker/ubuntu:act-22.04 plus a per-run download of yamllint, check-jsonschema, kyverno,
promtool and chainsaw. Local (dev.sh) and CI now gate on the identical toolchain.

- actions/checkout (Node) replaced by a plain-git step with fetch-depth-0 semantics; the
  toolbox has no Node. The job token is not persisted into .git/config.
- The install step is replaced by a check that every ci/versions.env pin equals the image's
  /etc/dev-toolchain-versions, so the pins stay enforced.
- Renovate follows the digest through a `# renovate-digest:` marker; github-actions manager
  disabled for own images.
- install-tools.sh kept (merge-gate.yaml blueprint + alert-rules-unit-tested check use it).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:12:03 -06:00
jakman85andClaude Opus 5.5 b548224db8 feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Skipped
The in-cluster runner ran at the default capacity 1 with no config file, so
gate / images / renovate runs on different refs queued behind each other
(29-759s waits; run 472 waited 759s behind a renovate burst).

- ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via
  CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault()
  applies the same defaults to a partial file as to no file; the
  generate-config example is deliberately not used (its labels would override
  the registered ones, force_pull differs from the default).
- Pod-template config-revision annotation so ConfigMap-only edits can roll it.
- dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi.
  Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs.
- runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1).

apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the
renderer only places it in a wave), so it is the source, not generated output.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:09:08 -06:00
jakman85andClaude Opus 5.5 759a0e51c8 chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
- Replace the deprecated top-level `docker` key with the packageRule Renovate's
  own migration produced (matchCategories docker, enabled true), keeping every
  comment. Supersedes #139, which dropped all comment lines.
- Add gitea/runner (act_runner's upstream rename) to the blast-radius rule.
- rangeStrategy bump for flux/helm `>=A <B` ranges: the default `auto` resolves
  to `replace`, which turned `>=1.12.0 <1.13.0` into `<1.14.0` (#131, #130,
  #115). In-range `bump` updates are disabled; Flux adopts those unaided.
- Restore headlamp's lower bound: `<0.47.0` -> `>=0.46.0 <0.47.0` (live 0.46.0).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:05:49 -06:00
jakman85andClaude Opus 5.5 ea30ba876c fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)
gate / gate (pull_request) Successful in 3m5s
gate / conformance (pull_request) Skipped
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2
ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new
operator fails to start its keycloaksamlclients informer (Not Found) and
CrashLoops, holding Flux `platform` in health checks.

- add both CRD URLs with the same renovate marker/depName
- patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was
  OOMKilled at 421/450Mi, 154 restarts)
- README/renovate comments: Renovate cannot add CRDs a release introduces

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:00:01 -06:00
jakman85andClaude Opus 5.5 3d7ecbcfe9 fix(conformance): regenerate L1 render goldens for oauth2-proxy v7.15.6
gate / gate (pull_request) Successful in 3m0s
gate / conformance (pull_request) Skipped
PR #136 bumps the renderer's OAUTH2_PROXY_IMAGE; the container-workload
and egress-class (vpn + internet) goldens still pinned v7.8.1. Regenerated
with the check scripts' own invocation; each diff is the image tag only.
Also updates the crafty README's current-pin row.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 23:58:29 -06:00
jakman85 e8d26669a4 Merge pull request 'chore(deps): Update dependency keycloak/keycloak-k8s-resources to v26.8.2' (#121) from renovate/keycloak-operator into main
gate / gate (push) Successful in 3m0s
gate / conformance (push) Skipped
Reviewed-on: #121
2026-11-08 05:53:29 +00:00
jakman85 048254bfaf Merge pull request 'feat(gitea): upgrade 1.27.3 -> 28.3.0' (#142) from feat/gitea-28 into main
gate / gate (push) Successful in 2m59s
gate / conformance (push) Skipped
Reviewed-on: #142
2026-11-08 05:41:34 +00:00
jakman85 dd94101647 Merge pull request 'feat(devcontainer): bake velero CLI into the toolbox, pinned to prod server v1.18.2' (#143) from feat/devcontainer-velero-cli into main
gate / gate (push) Successful in 3m0s
gate / conformance (push) Skipped
images / images (push) Successful in 38s
Reviewed-on: #143
2026-11-08 05:30:28 +00:00
jakman85andClaude Opus 5.5 60e3a24652 feat(devcontainer): bake velero CLI into the toolbox, pinned to prod server v1.18.2
gate / gate (pull_request) Successful in 3m1s
images / images (pull_request) Successful in 44s
gate / conformance (pull_request) Skipped
The velero CLI and server are version-coupled, so VELERO_VERSION follows the
server running on prod (docker.io/velero/velero:v1.18.2, chart 12.2.3,
verified 2026-11-07). It has no renovate marker on purpose: bump it together
with the chart's appVersion. The verify stage asserts that
`velero version --client-only` reports the pin. The build arg reaches CI
through build-image.sh --print-build-args, so no .gitea/ or ci/ change is
needed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 23:18:21 -06:00
jakman85andClaude Opus 5.5 8a9baa92bc feat(gitea): upgrade 1.27.3 -> 28.3.0
gate / gate (pull_request) Successful in 3m13s
gate / conformance (pull_request) Skipped
Gitea dropped the "1." prefix; 28.x is the successor of 1.27.
- image 28.3.0, update.range >=28.0.0 <29.0.0, Renovate ceiling <29.0.0
- drop GITEA__server__DOMAIN (no longer read in 28, ROOT_URL is the source)
- declare RUN_RETENTION_DAYS=400, EGRESS_MODE=lax (no allow list: the OIDC
  login source is not gated by the egress policy in 28.3.0), and
  DISABLE_REGISTRATION=true (28's new default)
- document autoRollbackOnFailedHealth as inert (no ImagePolicy is rendered);
  rollback is a PVC restore because the DB migration is one-way

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 22:55:06 -06:00
jakman85 2d32b9bd7d Merge pull request 'chore(deps): Update actions/upload-artifact action to v7' (#113) from renovate/major-github-artifact-actions into main
gate / gate (push) Successful in 3m14s
gate / conformance (push) Skipped
Reviewed-on: #113
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 04:52:04 +00:00
jakman85 9f82b2accf Merge pull request 'chore(deps): Update actions/checkout action to v7' (#112) from renovate/actions-checkout-7.x into main
gate / gate (push) Canceled after 3s
gate / conformance (push) Canceled after 0s
images / images (push) Successful in 4s
Reviewed-on: #112
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 04:48:40 +00:00
jakman85 990b309cbb Merge pull request 'chore(deps): Update docker Docker tag to v29' (#114) from renovate/docker-29.x into main
gate / gate (push) Canceled after 9m30s
gate / conformance (push) Canceled after 0s
Reviewed-on: #114
2026-11-08 04:39:04 +00:00
jakman85 cedcb01975 Merge pull request 'chore(deps): Update ghcr.io/cloudnative-pg/postgresql Docker tag to v16.15' (#124) from renovate/ghcr.io-cloudnative-pg-postgresql-16.x into main
gate / gate (push) Successful in 3m18s
gate / conformance (push) Skipped
Reviewed-on: #124
2026-11-08 04:28:19 +00:00
jakman85 2b279d4f21 Merge pull request 'chore(deps): Update Helm release headlamp to <0.47.0' (#130) from renovate/headlamp-0.x into main
gate / gate (push) Canceled after 15s
gate / conformance (push) Canceled after 0s
Reviewed-on: #130
2026-11-08 04:28:01 +00:00
jakman85 12c28adf4d Merge pull request 'feat(conformance): L0 guard — external-dns annotationPrefix must match the tree (stacked on #140)' (#141) from feat/l0-externaldns-annotation-prefix into main
gate / gate (push) Successful in 2m55s
gate / conformance (push) Skipped
Reviewed-on: #141
2026-11-08 00:08:29 +00:00
jakman85 b0230ba736 Merge pull request 'fix(dns): pin external-dns annotationPrefix — v0.23 ignored every alpha annotation' (#140) from fix/public-dns-wan-a-record into main
gate / gate (push) Successful in 2m53s
gate / conformance (push) Skipped
Reviewed-on: #140
2026-11-08 00:01:06 +00:00
jakman85andClaude Opus 5.5 9c51f60a35 feat(conformance): L0 guard that the external-dns annotationPrefix matches the tree
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
Add assertion (8) to internal-services-publish-no-public-dns. The external-dns
HelmRelease must declare annotationPrefix explicitly, and every
external-dns annotation in apps/, platform/, runbooks/ and catalog/render.py
must use that prefix. The v0.23.0 default-prefix change on 2026-11-07
ignored the controller: none, target and hostname annotations while
assertions (1)-(3) stayed green.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 17:56:12 -06:00
jakman85andClaude Opus 5.5 267446c68f fix(dns): pin external-dns annotationPrefix to the alpha prefix
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
external-dns v0.22 changed its default annotation prefix to
external-dns.kubernetes.io/ with no fallback. Renovate #129 moved the chart
to 1.23.0 (app v0.23.0), and at 12:13Z it ignored every alpha-prefixed
annotation in this repo: the Gateway target (public names went to A
192.168.2.20), controller: none (jackett, pihole and semaphore got
published), and the L4 Service hostnames (unifi and crafty-mc were deleted).

Set annotationPrefix: external-dns.alpha.kubernetes.io/ to restore the v0.21
reading. Floor the chart at 1.23.0, because the flag needs app >= 0.22.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 17:50:13 -06:00
jakman85 d1994b16e8 Merge pull request 'chore(deps): Update Helm release external-dns to <1.24.0' (#129) from renovate/external-dns-1.x into main
gate / gate (push) Successful in 2m57s
gate / conformance (push) Skipped
Reviewed-on: #129
2026-11-07 12:12:00 +00:00
jakman85 d7317606e7 Merge pull request 'feat: PM-Agent also need to look at PR comments' (#117) from feat/pm-agent-pr-mentions into main
gate / gate (push) Successful in 2m55s
gate / conformance (push) Skipped
images / images (push) Successful in 1m25s
Reviewed-on: #117
2026-11-07 12:07:34 +00:00
jakman85 970d65d193 Merge pull request 'chore(deps): Update python Docker tag to v3.15.2' (#135) from renovate/python-3.x into main
gate / gate (push) Canceled after 1m25s
gate / conformance (push) Canceled after 0s
Reviewed-on: #135
2026-11-07 12:06:07 +00:00
jakman85 6719082e5c Merge pull request 'chore(deps): Update gitea/act_runner Docker tag to v0.6.1' (#127) from renovate/gitea-act_runner-0.x into main
gate / gate (push) Canceled after 1m13s
gate / conformance (push) Canceled after 0s
Reviewed-on: #127
2026-11-07 12:04:54 +00:00
jakman85 428b77f08c Merge pull request 'chore(deps): Update gitea/gitea Docker tag to v1.27.3' (#128) from renovate/gitea-gitea-1.x into main
gate / gate (push) Successful in 2m59s
gate / conformance (push) Skipped
Reviewed-on: #128
2026-11-07 11:44:36 +00:00
renovate-bot 2ee7d5036a Merge pull request 'chore(deps): Update registry.k8s.io/sig-storage/snapshot-controller Docker tag to v8.7.0' (#138) from renovate/registry.k8s.io-sig-storage-snapshot-controller-8.x into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
2026-11-07 09:22:38 +00:00
renovate-bot 5736f42e22 Merge pull request 'chore(deps): Update registry.gitlab.com/crafty-controller/crafty-4 Docker tag to v4.11.0' (#137) from renovate/registry.gitlab.com-crafty-controller-crafty-4-4.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 09:19:37 +00:00
renovate-bot 773254e3a1 Merge pull request 'chore(deps): Update nginxinc/nginx-unprivileged Docker tag to v1.31.7' (#134) from renovate/nginxinc-nginx-unprivileged-1.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 09:10:44 +00:00
renovate-bot 264cd7a46d Merge pull request 'chore(deps): Update lscr.io/linuxserver/radarr Docker tag to v5.28.0' (#133) from renovate/lscr.io-linuxserver-radarr-5.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 09:07:45 +00:00
renovate-bot 979cc4b21d Merge pull request 'chore(deps): Update lscr.io/linuxserver/qbittorrent Docker tag to v5.3.0' (#132) from renovate/lscr.io-linuxserver-qbittorrent-5.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 09:04:44 +00:00
renovate-bot be6742c22b Merge pull request 'chore(deps): Update ghcr.io/seerr-team/seerr Docker tag to v3.5.1' (#126) from renovate/ghcr.io-seerr-team-seerr-3.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 08:46:56 +00:00
renovate-bot c8b915d8f2 Merge pull request 'chore(deps): Update ghcr.io/gethomepage/homepage Docker tag to v2.7.0' (#125) from renovate/ghcr.io-gethomepage-homepage-2.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 08:43:58 +00:00
renovate-bot 6cd3bd29a9 Merge pull request 'chore(deps): Update favonia/cloudflare-ddns Docker tag to v1.18.0' (#123) from renovate/favonia-cloudflare-ddns-1.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 08:37:58 +00:00
renovate-bot 2040d35222 Merge pull request 'chore(deps): Update docker.io/library/alpine Docker tag to v3.24' (#122) from renovate/docker.io-library-alpine-3.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 08:35:00 +00:00
renovate-bot 47fc27ff08 Merge pull request 'chore(deps): Update velero/velero-plugin-for-aws Docker tag to v1.14.4' (#120) from renovate/velero-velero-plugin-for-aws-1.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 08:29:02 +00:00
renovate-bot 5e8b080298 Merge pull request 'chore(deps): Update semaphoreui/semaphore Docker tag to v2.19.17' (#119) from renovate/semaphoreui-semaphore-2.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 08:26:05 +00:00
renovate-bot 24a0e1e4c5 Merge pull request 'chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3025-ls29' (#118) from renovate/lscr.io-linuxserver-jackett-0.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-07 08:23:06 +00:00
renovate-bot 3aec6886d8 chore(deps): Update docker Docker tag to v29
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:19:08 +00:00
renovate-bot 1b2ec82a9d chore(deps): Update actions/upload-artifact action to v7
gate / gate (pull_request) Successful in 3m24s
gate / conformance (pull_request) Skipped
2026-11-07 08:19:06 +00:00
renovate-bot 3724d3068b chore(deps): Update actions/checkout action to v7
gate / gate (pull_request) Successful in 3m38s
images / images (pull_request) Successful in 48s
gate / conformance (pull_request) Skipped
2026-11-07 08:19:03 +00:00
renovate-bot c6ad6b559c chore(deps): Update registry.k8s.io/sig-storage/snapshot-controller Docker tag to v8.7.0
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:19:01 +00:00
renovate-bot 6efc49d415 chore(deps): Update registry.gitlab.com/crafty-controller/crafty-4 Docker tag to v4.11.0
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:18:58 +00:00
renovate-bot 893c9dee4f chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6
gate / gate (pull_request) Failing after 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:18:51 +00:00
renovate-bot a6dc2d1adb chore(deps): Update python Docker tag to v3.15.2
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:18:46 +00:00
renovate-bot 75a472cf67 chore(deps): Update nginxinc/nginx-unprivileged Docker tag to v1.31.7
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:18:43 +00:00
renovate-bot 24b02f37df chore(deps): Update lscr.io/linuxserver/radarr Docker tag to v5.28.0
gate / gate (pull_request) Successful in 3m0s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:18:02 +00:00
renovate-bot be3ce4645b chore(deps): Update lscr.io/linuxserver/qbittorrent Docker tag to v5.3.0
gate / gate (pull_request) Successful in 3m1s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:42 +00:00
renovate-bot 619c2943da chore(deps): Update Helm release longhorn to <1.14.0
gate / gate (pull_request) Successful in 2m54s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:40 +00:00
renovate-bot b7a77a7d1e chore(deps): Update Helm release headlamp to <0.47.0
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:34 +00:00
renovate-bot 1a94b0535c chore(deps): Update Helm release external-dns to <1.24.0
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:30 +00:00
renovate-bot 0e7d0204b3 chore(deps): Update gitea/gitea Docker tag to v1.27.3
gate / gate (pull_request) Successful in 3m1s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:22 +00:00
renovate-bot b77572f843 chore(deps): Update gitea/act_runner Docker tag to v0.6.1
gate / gate (pull_request) Successful in 2m55s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:20 +00:00
renovate-bot 76b5d038fb chore(deps): Update ghcr.io/seerr-team/seerr Docker tag to v3.5.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:16 +00:00
renovate-bot 306f6f4fb0 chore(deps): Update ghcr.io/gethomepage/homepage Docker tag to v2.7.0
gate / gate (pull_request) Successful in 3m0s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:12 +00:00
renovate-bot 3ceaad15e0 chore(deps): Update ghcr.io/cloudnative-pg/postgresql Docker tag to v16.15
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:11 +00:00
renovate-bot ae772e050d chore(deps): Update favonia/cloudflare-ddns Docker tag to v1.18.0
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:08 +00:00
renovate-bot 517dfc9030 chore(deps): Update docker.io/library/alpine Docker tag to v3.24
gate / gate (pull_request) Successful in 3m0s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:05 +00:00
renovate-bot 0605ecf593 chore(deps): Update dependency keycloak/keycloak-k8s-resources to v26.8.2
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:02 +00:00
renovate-bot 16de787557 chore(deps): Update velero/velero-plugin-for-aws Docker tag to v1.14.4
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:16:58 +00:00
renovate-bot b7cb70955c chore(deps): Update semaphoreui/semaphore Docker tag to v2.19.17
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:16:49 +00:00
renovate-bot 9bcea04b1b chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3025-ls29
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:15:48 +00:00
pm-agent e8e9de8611 feat: PM-Agent also need to look at PR comments
gate / gate (pull_request) Successful in 2m58s
images / images (pull_request) Successful in 1m6s
gate / conformance (pull_request) Has been skipped
From Gitea issue #116.

Authored by the PM agent's worker; the merge gate is the approver. Refs #116.

Agent: pm-agent (issue #116, mode revise)
2026-11-07 06:02:58 +00:00
pm-agent 2fdb95393c feat: PM-Agent also need to look at PR comments
gate / gate (pull_request) Successful in 2m55s
images / images (pull_request) Successful in 1m10s
gate / conformance (pull_request) Has been skipped
From Gitea issue #116.

Authored by the PM agent's worker; the merge gate is the approver. Refs #116.

Agent: pm-agent (issue #116, mode implement)
2026-11-07 05:18:24 +00:00
jakman85 5b484f6b78 Merge pull request 'chore(deps): Update Helm release cilium to v1.20.4' (#104) from renovate/cilium-1.x into main
gate / gate (push) Successful in 2m48s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 4m59s
Reviewed-on: #104
2026-11-07 03:07:19 +00:00
renovate-bot 640edb758d Merge pull request 'chore(deps): Update plexinc/pms-docker Docker tag to v1.43.6.11085-d1e52e169' (#111) from renovate/plexinc-pms-docker-1.x into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m58s
2026-11-05 08:27:21 +00:00
renovate-bot e6cd064f5c Merge pull request 'chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3009-ls26' (#110) from renovate/lscr.io-linuxserver-jackett-0.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-05 08:24:23 +00:00
renovate-bot dd9d8da86a chore(deps): Update plexinc/pms-docker Docker tag to v1.43.6.11085-d1e52e169
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-05 08:17:16 +00:00
renovate-bot c3ce4db767 chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3009-ls26
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-05 08:16:08 +00:00
renovate-bot 38076f1cff chore(deps): Update Helm release cilium to v1.20.4
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-05 08:15:53 +00:00
renovate-bot 552330398b Merge pull request 'chore(deps): Update pihole/pihole Docker tag to v2026.10.1' (#109) from renovate/pihole-pihole-2026.x into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 3m19s
2026-11-04 08:22:16 +00:00
renovate-bot e49aebc803 Merge pull request 'chore(deps): Update docker.io/moby/buildkit Docker tag to v0.34.2' (#108) from renovate/docker.io-moby-buildkit-0.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-04 08:19:20 +00:00
renovate-bot 9c0292001e chore(deps): Update pihole/pihole Docker tag to v2026.10.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-04 08:16:12 +00:00
renovate-bot db9416d3b5 chore(deps): Update docker.io/moby/buildkit Docker tag to v0.34.2
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-04 08:15:52 +00:00
renovate-bot cb4a1bf2de Merge pull request 'chore(deps): Update lscr.io/linuxserver/sonarr Docker tag to v4.0.20' (#107) from renovate/lscr.io-linuxserver-sonarr-4.x into main
gate / gate (push) Successful in 3m3s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m12s
2026-11-03 08:22:33 +00:00
renovate-bot 028f5229f0 Merge pull request 'chore(deps): Update kubevirt-cdi' (#106) from renovate/kubevirt-cdi into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
2026-11-03 08:19:36 +00:00
renovate-bot 44e2cd3c64 chore(deps): Update lscr.io/linuxserver/sonarr Docker tag to v4.0.20
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-11-03 08:16:12 +00:00
renovate-bot b30b7e6f67 chore(deps): Update kubevirt-cdi
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-03 08:16:06 +00:00
renovate-bot 5a3a5f8df2 Merge pull request 'chore(deps): Update intel/intel-gpu-plugin Docker tag to v0.37.1' (#105) from renovate/intel-intel-gpu-plugin-0.x into main
gate / gate (push) Successful in 2m57s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m31s
2026-11-02 08:22:07 +00:00
renovate-bot f698bc82a0 chore(deps): Update intel/intel-gpu-plugin Docker tag to v0.37.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-02 08:16:06 +00:00
renovate-bot 6bc10fb60d Merge pull request 'chore(deps): Update dependency intel/intel-device-plugins-for-kubernetes to v0.37.1' (#102) from renovate/intel-intel-device-plugins-for-kubernetes-0.x into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m4s
2026-11-01 08:18:56 +00:00
renovate-bot d8c330fb8c chore(deps): Update dependency intel/intel-device-plugins-for-kubernetes to v0.37.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-01 08:15:52 +00:00
jakman85 a1d89f4a5f Merge pull request 'chore(pm-agent): pin agent-runner to the run-394 digest (466f63e, PR #99), bump config-generation' (#100) from chore/pm-agent-digest-99 into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 51s
Reviewed-on: #100
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 17:23:18 +00:00
jakman85andClaude Opus 5.5 19097e4d4f chore(pm-agent): pin agent-runner to the run-394 digest (466f63e, PR #99), bump config-generation
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
Dispatcher image and PM_WORKER_IMAGE both move to
sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the
registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 12:17:00 -05:00
jakman85 466f63ec3c Merge pull request 'fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS' (#99) from fix/pm-agent-triage-exit into main
gate / gate (push) Successful in 2m56s
images / images (push) Successful in 16s
gate / conformance (push) Has been skipped
Reviewed-on: #99
2026-10-31 17:12:40 +00:00
jakman85andClaude Opus 5.5 664ee82b9a fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS
gate / gate (pull_request) Successful in 2m56s
images / images (pull_request) Successful in 13s
gate / conformance (pull_request) Has been skipped
`claude --print --output-format json` reports its own failures (API error, usage limit,
error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s
stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31.
describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails,
redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged)
makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 12:04:09 -05:00
jakman85 95e08da2f0 Merge pull request 'chore(pm-agent): take the dispatcher out of dry run (operator-approved), config-generation 4' (#98) from chore/pm-agent-go-live into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
Reviewed-on: #98
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 16:52:16 +00:00
jakman85andClaude Opus 5.5 0d3e51acc4 chore(pm-agent): turn PM_DRY_RUN off (operator-approved), bump config-generation to 4
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry,
which admission requires, and the dispatcher's dry-run triage of #97 was read and approved.
PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the
configmap comments, the README and the plan's follow-up 3 is updated to match.

Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding
present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both
egress CNPs Valid.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:45:52 -05:00
jakman85 11267409d7 Merge pull request 'chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation' (#96) from chore/pm-agent-runner-digest into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
Reviewed-on: #96
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 16:34:49 +00:00
jakman85andClaude Opus 5.5 6a192ede2a chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule
refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry),
and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read
from the published layer).

Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's
dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The
renovate-digest markers are unchanged, so later bumps remain Renovate's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:22:52 -05:00
jakman85 06d4a9a1fc Merge pull request 'Added the credentials' (#95) from feature/pm-agent-credentials into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
Reviewed-on: #95
2026-10-31 16:20:12 +00:00
jakman85 9db20e8ca0 Added the credentials
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-10-31 11:11:22 -05:00
jakman85 ddd04d460e Merge pull request 'feat(pm-agent): pull credential + admission rule 7, first published digest pinned, dry-run first start, token scopes fixed' (#61) from feat/pm-agent-registry-pull-secret into main
gate / gate (push) Successful in 2m53s
images / images (push) Successful in 15s
gate / conformance (push) Has been skipped
Reviewed-on: #61
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 15:46:54 +00:00
jakman85 f0760d1024 Merge pull request 'feat(images): pin pihole, jackett and plex to the digests already running; Renovate follows them' (#94) from feat/pin-pihole-jackett-plex into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Reviewed-on: #94
2026-10-31 15:45:17 +00:00
jakman85 553d027f7c Merge branch 'main' into feat/pm-agent-registry-pull-secret
gate / gate (pull_request) Successful in 2m58s
images / images (pull_request) Successful in 34s
gate / conformance (pull_request) Has been skipped
2026-10-31 15:43:51 +00:00
jakman85andClaude Opus 5.5 183bca6897 feat(images): pin pihole, jackett and plex to the digests already running; Renovate follows them
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
pihole/pihole (no tag)               -> 2026.10.0               sha256:d3413471...
lscr.io/linuxserver/jackett:latest   -> v0.24.2865-ls7          sha256:4fcf3cc2...
plexinc/pms-docker:latest            -> 1.43.4.10903-e5521bd8c  sha256:e0ab2739...

Each tag was resolved against its registry and matched the pods' imageID, so the image is
unchanged (the pod template string does change, so Flux rolls the three workloads).

renovate.json5: regex versioning for jackett (-lsN) and plex (build-hex); jackett is looked
up on ghcr.io via overridePackageName, because lscr.io is capped at dockerMaxPages=20 pages
and jackett's 32k tags put the running one at #31,980 (dry run proposed nothing without it);
jackett weekly (13 builds in 12 days); pihole deliberately unheld, year rollover = major.
Stale "publishes only latest" prose fixed in renovate.json5 and three runbooks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 10:38:07 -05:00
jakman85 ce849032d5 Merge pull request 'chore(deps): Pin dependencies' (#89) from renovate/pin-dependencies into main
gate / gate (push) Successful in 2m55s
images / images (push) Successful in 5s
gate / conformance (push) Has been skipped
Reviewed-on: #89
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 15:35:30 +00:00
renovate-bot b766130b85 Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 65cc7c1' (#92) from renovate/gitea.home.keithjackson.dev-home-network-conformance-runner-main into main
gate / gate (push) Successful in 2m57s
gate / conformance (push) Has been skipped
2026-10-31 15:26:58 +00:00
renovate-bot 862773f31b Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/toolbox:main Docker digest to d435fab' (#93) from renovate/gitea.home.keithjackson.dev-home-network-toolbox-main into main
gate / gate (push) Successful in 2m55s
images / images (push) Successful in 47s
gate / conformance (push) Has been skipped
2026-10-31 08:25:11 +00:00
renovate-bot db4ba79a1c Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to c4eba5b' (#90) from renovate/gitea.home.keithjackson.dev-home-network-agent-runner-main into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-10-31 08:22:14 +00:00
renovate-bot 0b6282aba4 chore(deps): Update gitea.home.keithjackson.dev/home-network/toolbox:main Docker digest to d435fab
gate / gate (pull_request) Successful in 2m57s
images / images (pull_request) Successful in 5s
gate / conformance (pull_request) Has been skipped
2026-10-31 08:16:11 +00:00
renovate-bot c6cfaf62b6 chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 65cc7c1
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
2026-10-31 08:16:10 +00:00
renovate-bot 4ae648f6d3 chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to c4eba5b
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
2026-10-31 08:16:08 +00:00
jakman85 a4d5206294 Merge pull request 'fix(renovate): never auto-merge the oauth2-proxy sidecar or Renovate's own version' (#87) from fix/renovate-hold-auth-and-self into main
gate / gate (push) Successful in 3m15s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m8s
Reviewed-on: #87
2026-10-30 08:21:22 +00:00
renovate-bot 47eb6afa02 chore(deps): Pin dependencies
gate / gate (pull_request) Successful in 3m8s
images / images (pull_request) Successful in 1m41s
gate / conformance (pull_request) Has been skipped
2026-10-30 08:20:40 +00:00
jakman85 74ba90c0b3 Merge branch 'main' into fix/renovate-hold-auth-and-self
gate / gate (pull_request) Successful in 2m55s
gate / conformance (pull_request) Has been skipped
2026-10-30 08:14:15 +00:00
jakman85 fb45d7441b Merge pull request 'feat(renovate): track the Keycloak operator (the real Keycloak version), never auto-merge it' (#88) from feat/renovate-track-keycloak-operator into main
gate / gate (push) Successful in 2m54s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 50s
Reviewed-on: #88
2026-10-30 08:14:02 +00:00
jakman85andClaude Opus 5.5 8ed4e30201 docs(decisions): handoff for feat/renovate-track-keycloak-operator
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 03:06:56 -05:00
jakman85andClaude Opus 5.5 24db269ea8 feat(renovate): track the Keycloak operator (the real Keycloak version), never auto-merge it
The Keycloak CR has no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the
server image; the three keycloak-k8s-resources release URLs in
platform/keycloak/kustomization.yaml are therefore THE Keycloak version, and Renovate
could not see them (no `# renovate:` marker). They now carry markers under one depName
(one branch, groupName keycloak-operator), with a packageRule: automerge false, labels
needs-review + blast-radius; majors still wait on the dashboard.

The untagged quay.io/keycloak/keycloak in catalog/keycloak.yaml renders
apps/wave-1/keycloak.yaml, which no kustomization lists. It is pinned to the operator's
26.0.6 as documentation, re-rendered, and Renovate is disabled for that image so it
cannot raise PRs against a file nothing deploys. The render is kept (not deleted): it is
the fixture for apps/wave-1/tests/app-slis.test.yaml's keycloak SLI rule tests.

README: an "Upgrading Keycloak" section — an operator bump must be checked against
passkey-reconciler.yaml's 26.0.6 Admin API assumptions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 03:02:25 -05:00
jakman85andClaude Opus 5.5 df96be98bf fix(renovate): never auto-merge the oauth2-proxy sidecar or Renovate's own version
gate / gate (pull_request) Successful in 2m51s
gate / conformance (pull_request) Has been skipped
Operator decision. Both are held at every update type and labelled needs-review:

- quay.io/oauth2-proxy/oauth2-proxy (render.py constant, rendered sidecars, crafty). Since v7.13.0
  oauth2-proxy validates refreshed OIDC sessions against the access token, every sidecar uses
  --cookie-refresh, and that path is untested against this Keycloak.
- ghcr.io/renovatebot/renovate (and renovatebot/github-action if ever reintroduced). Nothing
  re-validates renovate.json5 against the new version before it takes effect.

The validator passes on 44.167.0. Evaluated with Renovate's own packageRules engine: exactly 2 of 29
branch rows change (automerge true -> false, +needs-review); the other 27 are identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:54:51 -05:00
jakman85 b4ff2879be Merge pull request 'fix(renovate): edit catalog sources, track CNPG imageName, honour update.range, guard real-run auto-merges' (#86) from fix/renovate-run-hygiene into main
gate / gate (push) Successful in 2m53s
gate / conformance (push) Has been skipped
Reviewed-on: #86
2026-10-30 07:43:28 +00:00
jakman85andClaude Opus 5.5 d9571fc6d8 fix(renovate): edit catalog sources, track CNPG imageName, honour update.range, guard real-run auto-merges
gate / gate (pull_request) Successful in 2m55s
gate / conformance (pull_request) Has been skipped
Audit of the first completed dry runs (353/354) plus a debug-level local reproduction:

- App-image updates edited only the generated apps/wave-*. A ServiceDefinition has no apiVersion,
  so the kubernetes manager extracts nothing from catalog/, and every such PR would fail
  render-golden (reproduced: GOLDEN DRIFT). New regex managers for catalog source.image and
  render.py OAUTH2_PROXY_IMAGE put the source and generated edits in one branch (render-golden
  rc 0, reproduced).
- CNPG Cluster imageName was untracked (3 Clusters), while the grant Job was tracked. New manager.
- allowedVersions mirror catalog update.range: this drops gitea 28.x, qbittorrent 20.x, radarr 6,
  nextcloud 35 and postgresql 18.
- Never auto-merge: gitea, act_runner, postgresql (blast radius), flux range widening, python
  minors. Majors wait for a Dependency Dashboard tick.
- velero plugin RC: semver versioning (v1.14.3-rc.2 -> v1.14.4 now proposed).
- flux docker duplicates disabled (the "Cannot find replaceString" line). Two false comments
  corrected.

Local dry run: regex deps 7 -> 20, branches 33 -> 29. The validator passes on 44.167.0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:35:33 -05:00
jakman85 593d0fad15 Merge pull request 'fix(renovate): document and probe the five token scopes Renovate needs; lint daemon-bound actions' (#85) from fix/renovate-auth into main
gate / gate (push) Successful in 2m54s
gate / conformance (push) Has been skipped
Reviewed-on: #85
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 07:10:02 +00:00
jakman85andClaude Opus 5.5 0c6ee4921d fix(renovate): document and probe the five token scopes Renovate needs; lint daemon-bound actions
gate / gate (pull_request) Successful in 2m54s
gate / conformance (pull_request) Has been skipped
Run 350 cleared platform init (/user 200, /version 200, /repos 200), then stopped at
GET /api/v1/orgs/Home-Network with Gitea's own 403:
  token does not have at least one of required scope(s), required=[read:organization],
  token scope=read:package,write:repository,read:user
The workflow's header listed only those three scopes, and the operator minted exactly that.
Renovate's initRepo -> isOrg needs read:organization, and Gitea files /issues and /labels under
the issue scope, which the dashboard and PR labels need: write:issue.

- renovate.yaml: corrects the scope list (read:user, read:organization, write:repository,
  write:issue, read:package), with the call each serves. A new preflight step probes them with
  GETs, the token passed to curl on stdin, and names a missing scope instead of Renovate's bare
  "Authentication failure".
- ci/lint.sh: rejects workflow `uses:` lines that need a Docker daemon (docker://,
  renovatebot/github-action, docker/*-action), the class fixed in #84.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:55:39 -05:00
jakman85 d550a501e2 Merge pull request 'fix(renovate): run Renovate as the job container; the action needs a Docker daemon jobs do not get' (#84) from fix/renovate-first-run into main
gate / gate (push) Successful in 2m52s
gate / conformance (push) Has been skipped
Reviewed-on: #84
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 06:36:11 +00:00
jakman85andClaude Opus 5.5 365e43cdea fix(renovate): run Renovate as the job container; the action needs a Docker daemon jobs do not get
gate / gate (pull_request) Successful in 2m52s
gate / conformance (pull_request) Has been skipped
renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:

  failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory

The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.

Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:23:36 -05:00
jakman85andClaude Opus 5.5 7f0cbc7ba2 feat(pm-agent): pin the first published agent-runner digest, start dry, and fix the token scopes
gate / gate (pull_request) Failing after 2m56s
images / images (pull_request) Successful in 32s
gate / conformance (pull_request) Has been skipped
Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git
blockers to going live go away:

- deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3...,
  tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged.
- PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not
  just cautious: the run-340 image's baked worker template predates imagePullSecrets, so
  admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps.
- Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under
  read:user. The documented list was missing it. Now read:user, write:repository,
  write:issue, read:package everywhere.
- Org-package read documented as a team (Packages=Read, no repos), not just a scope.
- sops is toolbox-only; the .examples say so and give the in-place edit.
- runbook: key-name check no longer emits Secret values before the pipe.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:16:21 -05:00
jakman85andClaude Opus 5 a7eadd4054 feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.

pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.

ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.

The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.

kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.

The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-30 01:09:39 -05:00
jakman85 dfc432c6ea Merge pull request 'fix(keycloak-passkey): passkey reconciler with review fixes — bind only on git's say-so (default false), protect real sub-flows, body-id screening, failure alert' (#82) from feat/keycloak-passkey-reconcile into main
gate / gate (push) Successful in 2m50s
gate / conformance (push) Has been skipped
Reviewed-on: #82
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 05:59:11 +00:00
jakman85 262c6e4a97 Merge branch 'main' into feat/keycloak-passkey-reconcile
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-10-30 05:56:00 +00:00
jakman85 523cd9118f Merge pull request 'fix(buildkit): allow buildkitd egress to Gitea so the Gateway stops 403ing the registry push' (#83) from fix/gitea-admit-buildkit into main
gate / gate (push) Successful in 1m55s
gate / conformance (push) Has been skipped
Reviewed-on: #83
2026-10-30 05:50:12 +00:00
jakman85andClaude Opus 5.5 d2c4bd5194 fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 00:38:54 -05:00
jakman85 0dd57fbaf0 Merge pull request 'feat(images): build, verify and publish the three images in CI on rootless BuildKit; Renovate carries the digests' (#81) from feat/image-pipeline into main
gate / gate (push) Successful in 1m57s
gate / conformance (push) Has been skipped
images / images (push) Failing after 5s
Reviewed-on: #81
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 05:15:16 +00:00
jakman85andClaude Opus 5.5 04eb38584a fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal
gate / gate (pull_request) Successful in 1m58s
images / images (pull_request) Failing after 7s
gate / conformance (pull_request) Has been skipped
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 f65c12b395 feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.

- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
  digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
  read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
  :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
  No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
  `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
  the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
  targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
  Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 8f35234fcc fix(images): name the measured runner blocker; the job container gets no Docker daemon
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 e539109830 feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.

- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
  builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
  sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
  (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
  manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
  current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
  Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85 27da5555a4 Merge pull request 'fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT' (#80) from fix/renovate-config-filename into main
gate / gate (push) Successful in 1m54s
gate / conformance (push) Has been skipped
Reviewed-on: #80
Reviewed-by: Keith Jackson <[email protected]>
2026-10-30 04:48:42 +00:00
jakman85andClaude Opus 5.5 da1b36eb64 fix(renovate): pin renovate-version — the action's default (39) rejects renovate.json5
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:49:38 -05:00
jakman85andClaude Opus 5.5 6bb8af1d70 docs(decisions): handoff for the passkey reconciler review fixes
gate / gate (pull_request) Failing after 2m55s
gate / conformance (pull_request) Has been skipped
Also states in bind_browser_flow()'s docstring that it is reached only when git's
bind-browser-flow is "true".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:45:40 -05:00
jakman85andClaude Opus 5.5 7c4eb439b2 feat(monitoring): alert when the keycloak passkey reconciler's Job fails
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:41:15 -05:00
jakman85andClaude Opus 5.5 17eed6e8ae fix(keycloak-passkey): bind only on git's say-so; protect real sub-flows; screen the requirement PUT by body id
Review findings on the passkey reconciler, fixed:

1. HIGH - the bind was automatic and rollback did not stick. A git-declared switch,
   `bind-browser-flow` in passkey-desired-state.yaml, now decides whether pass 3 may
   PUT browserFlow. Default "false": build + verify the candidate (or reuse a verified
   one), log WOULD BIND <alias>, never touch the binding - including after an earlier
   bind, so an operator's rollback sticks, and a hand-edited bound candidate is left
   alone rather than shadowed by an -r2 build. "true" binds as before. The renderer
   carries the one hand-edited line through and --check refuses anything but the
   quoted strings "true"/"false". --no-bind can only narrow it.

2. HIGH - the protected-subtree guard read `alias` on sub-flow rows. Keycloak 26.0.6
   AuthenticationManagementResource.recurseExecutions sets displayName =
   flowRef.getAlias() and flowId on a sub-flow row and never sets alias (alias is the
   CONFIG alias on authenticator rows), so the protected set was only the top-level
   alias. Sub-flows are now named by displayName (getFlows() lists top-level flows
   only, so flowId cannot be resolved through it); a sub-flow row without one leaves
   the subtree undetermined and the flow pass fail-closed. The selftest stub now
   returns 26.0.6-shaped rows, and a real run is asserted to refuse writes into
   browser-mfa-forms and browser-mfa-conditional-otp.

3. MEDIUM - PUT /flows/{alias}/executions was screened on the path alias only, but
   updateExecutions resolves the execution realm-wide by body id. body.id is now
   screened against the protected ids AND required to be in the tree this run is
   building (recorded from the build's own readback); the config POST is held to the
   same build scope; and the body may carry only id/requirement/priority (the same
   handler renames any flow named by flowId).

4. Stale "audit-only" docs rewritten (platform-config kustomization, chainsaw test,
   desired-state header and its renderer template, README) to describe what ships.

5. backoffLimit 0 (a retried deterministic build failure burns the candidate-alias
   budget); the stale "Renovate does not manage image: tags" comment corrected and the
   image pinned by index digest.

6. The ~160-line header duplicated in the YAML and the script is now kept once, in the
   script, with a short pointer above it.

Selftest: 259 assertions (was 216). mutations.py: 40 injected regressions (was 30),
including one per finding above, all caught.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:41:14 -05:00
jakman85andClaude Opus 5 f5931944fa feat(keycloak): reconcile the passkey realm config via GitOps, the browser flow blue/green
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.

A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.

  pass 1  webAuthnPolicyPasswordless* -- strict read-modify-write of
          GET/PUT /admin/realms/homelab, readback-verified.
  pass 2  webauthn-register-passwordless -- strict read-modify-write of ONE
          literally-named alias; enabled, defaultAction false so enrolment stays
          opt-in.
  pass 3  the browser flow -- BLUE/GREEN.

PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:

  1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
     shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
     mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
     and zero user-visible effect.
  2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
     conditional-OTP config CONTENTS -- before anything is bound, so the
     regression that once 403'd every platform-admin is caught BEFORE it bites.
  3. BIND with ONE atomic PUT setting browserFlow.
  4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
     flow survives as the rollback target.

The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.

Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.

Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.

Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.

Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-29 21:22:08 -05:00
jakman85andClaude Opus 5.5 aacf361bb8 fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT
gate / gate (pull_request) Failing after 1m58s
gate / conformance (pull_request) Has been skipped
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.

Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 20:58:15 -05:00
jakman85 1301af9a4c Merge pull request 'fix(renovate): the config was INVALID, so the first manual run errored out' (#79) from fix/renovate-invalid-config into main
gate / gate (push) Successful in 1m55s
gate / conformance (push) Has been skipped
renovate / renovate (push) Failing after 9s
Reviewed-on: #79
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:29:34 +00:00
jakman85 42ce001d71 Merge pull request 'fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR' (#77) from fix/renovate-cron-window into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
Reviewed-on: #77
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:28:59 +00:00
jakman85 84bb0f0acd Merge pull request 'fix(pihole): make the split-horizon answer cacheable (local-ttl=60)' (#78) from fix/pihole-local-ttl into main
gate / gate (push) Successful in 1m53s
gate / conformance (push) Has been skipped
Reviewed-on: #78
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 08:19:28 +00:00
jakman85andClaude Opus 5 0d4bd7297d fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.

  03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
  window     ->  03:00-06:00 local   =>  NOT INSIDE (3h45m early)

The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.

Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
  EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
  EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.

Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.

STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:12:28 -05:00
jakman85andClaude Opus 5 713b0c3dba fix(renovate): the config was INVALID, so the first manual run errored out
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
The operator ran the Renovate workflow manually and it failed. The cause is not
the workflow: renovate.json was never a valid Renovate config. It carried five
pseudo-comment keys (_comment, _kubernetes_comment, _unmanageable_comment,
_ignorePaths_comment, _docker_comment) because JSON has no comments, and
Renovate REJECTS unknown top-level options. Proven with the official validator:

  ERROR: Found errors in configuration
    Invalid configuration option: _comment
    Invalid configuration option: _docker_comment
    Invalid configuration option: _ignorePaths_comment
    Invalid configuration option: _kubernetes_comment
    Invalid configuration option: _unmanageable_comment

This is the SECOND independent reason Renovate never worked, on top of PR #77's
cron firing outside renovate.json's own schedule window. Either alone was enough
to produce "nothing ever happens"; #73 shipped both.

Converted renovate.json -> renovate.json5. JSON5 permits real // comments and
Renovate discovers renovate.json5 natively; nothing in .gitea/ or ci/ pins the
old filename and no RENOVATE_CONFIG_FILE is set. Every word of the original
rationale is preserved as positioned comments (74 comment lines) rather than
flattened into a description string.

Verified the conversion dropped nothing: 18 non-comment keys before and after,
no missing/extra keys, no changed values, docker: {enabled: true} still present.
The validator now returns "Config validated successfully".

DISPROVED while here: `enabledManagers` containing bare `regex` was suspected of
needing to be `custom.regex` (which would silently disable every inline
`# renovate:` marker). The validator accepts `regex`; unfounded, unchanged.

Renovate has still never completed a run -- this fixes one of two blockers and
nothing in the gate catches an invalid config. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:10:25 -05:00
jakman85andClaude Opus 5 fda5c31a94 fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 02:56:49 -05:00
204 changed files with 18585 additions and 878 deletions

No files matched your search

+47 -1
View File
@@ -115,14 +115,60 @@ CI. That is a real coverage gap, not a phase.
| variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
| secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
## How images are built
This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
verified and published by [`images.yaml`](images.yaml), never from a workstation
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
builds, let alone pushes, the image. Build args come from each image's own
`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
proof exist once.
| Trigger | What happens |
|---|---|
| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
token would not trigger the required gate, which is why it is Renovate's job and not this one's.
Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
action, keeps every digest git pins plus the newest tags.
Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
`platform/buildkit` is deployed and its client certificate copied
(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
## Files
```
.gitea/workflows/
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
images.yaml build/verify/publish this repo's container images (see above)
scripts/
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
install-tools.sh installs the pinned tools per stage (from ci/versions.env) — used by the
dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
decide.sh the R-TEST-03 merge decision over job results
images.sh the image pipeline's driver (what to build, base, push, digest)
README.md this file
```
+100 -11
View File
@@ -3,8 +3,10 @@
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
# below for why the script, not `make`, is invoked here). It runs every stage in order —
# (R-TEST-02) — and since 2026-11-08 it runs INSIDE the same image: the job `container:` is the
# CI-built devcontainer toolbox (see "THE JOB IMAGE" below), not a generic runner image that
# downloads the pinned tools on every run. (See further below for why the script, not `make`,
# is invoked here.) It runs every stage in order —
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
# R-TEST-03 decision, printing
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
@@ -17,7 +19,8 @@
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
# environment it is:
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
# baked into the job image and checked against ci/versions.env two steps up.
# GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
@@ -41,6 +44,32 @@
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
# is retained dormant in merge-gate.yaml.
#
# THE JOB IMAGE (2026-11-08, feat/gate-toolbox-container). The gate job runs in
# gitea.home.keithjackson.dev/home-network/toolbox, pinned BY DIGEST — the image
# .gitea/workflows/images.yaml builds from infra/devcontainer/Containerfile, verifies, and
# publishes. It is the SAME image `infra/devcontainer/dev.sh` runs, so a local gate and the CI
# gate now execute the identical toolchain (R-TEST-02), not two installs that agree by intent.
# Before this, the job ran in ghcr.io/catthehacker/ubuntu:act-22.04 and spent ~11s of every run
# re-downloading yamllint, check-jsonschema, kyverno, promtool and chainsaw
# (.gitea/workflows/scripts/install-tools.sh l0 + l1; measured on runs 494/495).
# * PINNED BY DIGEST, MOVED BY RENOVATE. The `# renovate-digest:` marker above the image line
# is the repo-wide convention (renovate.json5, "This repo's own CI-published images"):
# Renovate follows the `main` tag and opens a digest-bump PR, whose OWN gate run then
# executes on the new image — the bump tests itself. infra/gitea/registry-retention.sh reads
# the same marker, so the pinned digest is never pruned. Never hand-edit it to a tag.
# * PULL AUTH: none needed. The Home-Network org is public and the registry issues an
# anonymous pull token (measured 2026-11-08: anonymous /v2/token -> manifest 200), so the
# runner's DinD pulls it like any public image. If the org is ever made private, add
# `credentials:` here with a read:package token secret — the automatic Actions token has no
# package scope at /v2/token (see images.yaml's header).
# * COLD PULL: ~467 MB compressed (1.54 GB unpacked). The DinD store is an emptyDir, so the
# first job after a runner-pod restart or a digest bump pays one pull; every other run finds
# it cached (forcePull=false), exactly as it already did for the catthehacker image.
# * NO NODE in the toolbox, so `actions/checkout` (a Node action) cannot run here. The checkout
# step is plain git instead — see its comment for how it keeps fetch-depth-0 semantics.
# * install-tools.sh is NOT deleted: merge-gate.yaml (the dormant blueprint) still calls it,
# and conformance/tests/l0-static/alert-rules-unit-tested asserts its promtool wiring.
---
name: gate
@@ -78,16 +107,76 @@ jobs:
# Single required status check: the static merge gate.
gate:
runs-on: ubuntu-latest
# The CI-built toolbox — the same image dev.sh runs. See "THE JOB IMAGE" in the header.
container:
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0 # full history so ci/touched-requirements.sh can diff vs main
- name: Install pinned CI toolchain (ci/versions.env)
# Plain git, because actions/checkout is a Node action and the toolbox has no Node. It
# reproduces what actions/checkout@v7 with `fetch-depth: 0` did here (read from its logged
# commands on runs 490/495): full history of every branch + tags, so
# ci/touched-requirements.sh can diff against origin/<base>; the exact event commit
# checked out (detached for a PR, as `-B <branch>` for a push). One difference, on
# purpose: the job token is passed to the single fetch through GIT_CONFIG_* environment
# variables and is NOT persisted into .git/config — nothing after this step needs
# authenticated git (ci/ and conformance/ make no network git calls), so there is no
# reason to leave a credential in the workspace for the gate's scripts to read.
- name: Checkout (git, full history)
env:
GIT_TOKEN: ${{ github.token }}
SERVER_URL: ${{ github.server_url }}
REPOSITORY: ${{ github.repository }}
SHA: ${{ github.sha }}
REF: ${{ github.ref }}
run: |
./.gitea/workflows/scripts/install-tools.sh l0
./.gitea/workflows/scripts/install-tools.sh l1
set -euo pipefail
# act runs steps with umask 0000, so without this every checked-out directory is
# world-writable — and Ansible then IGNORES infra/ansible/ansible.cfg as a source
# ("Ansible is being run in a world writable directory"), which made the L0 check
# prod-baremetal-not-dev-surface fail on this PR's first CI run (run 500). The
# toolbox has Ansible, so that check now drives it for real. 0022 = what a
# workstation clone gets (reproduced locally: 0000 fails the check, 0022 passes).
umask 0022
git config --global --add safe.directory "$PWD"
git init -q .
git remote add origin "${SERVER_URL}/${REPOSITORY}"
auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
git -c protocol.version=2 fetch --quiet --no-tags --prune --no-recurse-submodules origin \
'+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' \
"+${SHA}:refs/remotes/checkout/head"
unset auth GIT_TOKEN
case "$REF" in
refs/heads/*) git checkout -q --force -B "${REF#refs/heads/}" "$SHA" ;;
*) git checkout -q --force --detach "$SHA" ;;
esac
[ "$(git rev-parse HEAD)" = "$SHA" ] || { echo "::error::HEAD is not ${SHA}" >&2; exit 1; }
echo "checked out $(git log -1 --format='%H %s') (ref ${REF}); $(git rev-list --count HEAD) commits of history"
# The image is the toolchain, so prove it is the PINNED one before trusting it: every
# *_VERSION in ci/versions.env must equal what the image recorded at build time
# (/etc/dev-toolchain-versions, written by infra/devcontainer/Containerfile; its `verify`
# stage already asserted the version-sensitive binaries report those pins). This replaces
# the per-run install: a drifted toolbox fails HERE, naming the tool, instead of gating with
# a different engine. ci/versions.env and infra/devcontainer/versions.env must move together.
- name: Toolchain is the one ci/versions.env pins
run: |
set -euo pipefail
. ci/versions.env
drift=0
for var in $(grep -oE '^[A-Z_]+_VERSION' ci/versions.env); do
tool="$(printf '%s' "${var%_VERSION}" | tr 'A-Z_' 'a-z-')"
want="${!var}"
got="$(sed -n "s/^${tool}=//p" /etc/dev-toolchain-versions)"
if [ "$got" = "$want" ]; then
printf 'ok %-18s %s\n' "$tool" "$want"
else
printf '::error::DRIFT %s: ci/versions.env pins %s, the job image has %s\n' "$tool" "$want" "${got:-nothing}"
drift=1
fi
done
[ "$drift" -eq 0 ]
# R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
# spec-immutability check, so its result is a distinct, tolerant status line in the
+185
View File
@@ -0,0 +1,185 @@
# Container images — built, verified and published by CI, never from a workstation.
#
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
# one-off build/push from a workstation." Before this workflow, three images existed only as
# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
#
# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
# pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
# push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
# dispatch -> same as a push when run on main; verify-only on any other ref.
# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
#
# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
# 1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
# REGISTRY returns in the job summary.
# 2. Renovate (renovate.yaml) follows the `main` tag of each image through the
# `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
# 3. That PR goes through the gate like any other change, and Flux deploys it.
# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
# user's.
#
# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
# * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
# `gitea-actions` (ActionsUserID -2) and sets NO token scope;
# * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
# docker client uses) then requires read:package / write:package / all scope and answers 401;
# * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
# and no branch for that user, so even past login it would get at most READ on a public org.
# So: a personal access token with scope `write:package`, minted by an account that has package
# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
# bot account in a packages-only team is the least-privilege choice. The username sent with it is
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
# is optional and only labels the login.
#
# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
# (not privileged; uid 1000 in a user namespace), reached over mTLS at
# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
# Pull requests get the client certificate (they must build) but never a registry credential.
#
# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
#
# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
#
name: images
on:
pull_request:
branches: [main]
paths:
- infra/devcontainer/Containerfile
- infra/devcontainer/versions.env
- infra/devcontainer/build-image.sh
- infra/agent/**
- conformance/incluster/Containerfile
- conformance/incluster/conformance-clone
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
push:
branches: [main]
paths:
- infra/devcontainer/Containerfile
- infra/devcontainer/versions.env
- infra/devcontainer/build-image.sh
- infra/agent/**
- conformance/incluster/Containerfile
- conformance/incluster/conformance-clone
- conformance/incluster/build-image.sh
- .gitea/workflows/images.yaml
- .gitea/workflows/scripts/images.sh
- platform/buildkit/versions.env
workflow_dispatch:
inputs:
images:
description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
required: false
default: "all"
concurrency:
# Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
# recoverable (a rerun skips what is already published) but pointless to provoke.
group: images-${{ github.ref }}
cancel-in-progress: false
jobs:
images:
runs-on: ubuntu-latest
# Measured on the PR that introduced this file — see its decisions entry. Generous because a
# cold DinD store downloads the Fedora base and every pinned tool on each run.
timeout-minutes: 75
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
fetch-depth: 0 # the per-image change detection diffs against the base commit
# ── pull requests: build + verify, NO credential ───────────────────────────────────────
- name: Build and verify (pull request — nothing is pushed)
if: github.event_name == 'pull_request'
env:
MODE: verify
IMAGES: auto
PR_BASE_REF: ${{ github.base_ref }}
# The BUILDER's client certificate (platform/buildkit). Not a registry credential:
# a pull request can build, never push.
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: |
set -euo pipefail
BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
export BASE_SHA
./.gitea/workflows/scripts/images.sh
# ── non-main dispatch: verify only ──────────────────────────────────────────────────────
- name: Build and verify (dispatch off main — nothing is pushed)
if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
env:
MODE: verify
IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
# ── main: publish ───────────────────────────────────────────────────────────────────────
- name: Preflight - the push token must exist, and its absence must be LOUD
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
env:
REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
run: |
set -euo pipefail
if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
cat >&2 <<'MSG'
Images cannot be published without a registry credential, and this step fails rather
than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
make "nothing was published" look exactly like "published". That is how this repo went
a month with images that existed only on one workstation.
WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
registry's /v2/token exchange (routers/api/packages/container/container.go,
Authenticate) — see the header of .gitea/workflows/images.yaml.
TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
bot in a packages-only team), create an access token with scope `write:package` and
nothing else, then add it to this repository as the Actions secret
REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
account's name. Do not commit the token anywhere. Then re-run this workflow
(workflow_dispatch on main, images: all).
MSG
exit 1
fi
echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
- name: Build, verify and publish (main)
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
env:
MODE: publish
IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
BASE_SHA: ${{ github.event.before }}
REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
run: ./.gitea/workflows/scripts/images.sh
+10 -10
View File
@@ -60,9 +60,9 @@ jobs:
spec-immutability:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4 # the implementation repo
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 # the implementation repo
- name: Check out the read-only spec (sibling path)
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
repository: ${{ vars.SPEC_REPO_SLUG }} # e.g. Home-Network/home-network-k3s (set in Gitea)
path: ../home-network-k3s
@@ -74,7 +74,7 @@ jobs:
runs-on: ubuntu-latest
needs: [spec-immutability]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with: { fetch-depth: 0 } # full history so touched-requirement diff works
- name: Install pinned tools
run: ./.gitea/workflows/scripts/install-tools.sh lint
@@ -85,7 +85,7 @@ jobs:
runs-on: ubuntu-latest
needs: [lint]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with: { fetch-depth: 0 }
- name: Install pinned tools
run: ./.gitea/workflows/scripts/install-tools.sh l0
@@ -96,7 +96,7 @@ jobs:
runs-on: ubuntu-latest
needs: [l0]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with: { fetch-depth: 0 }
- name: Install pinned tools
run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -107,7 +107,7 @@ jobs:
runs-on: ubuntu-latest
needs: [render-golden]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with: { fetch-depth: 0 }
- name: Install pinned tools
run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -115,7 +115,7 @@ jobs:
run: make -C ci l1
- name: Upload L1 artifacts
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: l1-junit-coverage
path: |
@@ -131,7 +131,7 @@ jobs:
needs: [l1]
if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with: { fetch-depth: 0 }
- name: Install pinned tools
run: ./.gitea/workflows/scripts/install-tools.sh conformance
@@ -142,7 +142,7 @@ jobs:
run: make -C ci conformance
- name: Upload L2/L3 artifacts
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: conformance-junit-coverage
path: |
@@ -160,7 +160,7 @@ jobs:
needs: [lint, l0, render-golden, l1, conformance]
if: always()
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- name: Decide
env:
R_LINT: ${{ needs.lint.result }}
+166 -21
View File
@@ -60,8 +60,21 @@
# entry's "What I did NOT verify" — that gap is stated, not papered over.
#
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
# `RENOVATE_TOKEN` — a Gitea access token for the bot account (`renovate-bot`) with FIVE scopes.
# Each is required by a call Renovate 44.x makes against Gitea 1.24.3, and Gitea refuses a
# token missing one with `403 token does not have at least one of required scope(s)`:
# read:user GET /user (platform init: initPlatform)
# read:organization GET /orgs/{owner}, /orgs/{o}/labels (initRepo -> isOrg, then org labels)
# write:repository clone/push branches, /pulls, /statuses
# write:issue /issues, /labels, /issues/{n}/labels (dependency dashboard, PR labels;
# Gitea files all of these under the ISSUE scope, not repository)
# read:package the Gitea container registry (RENOVATE_HOST_RULES below)
# CORRECTED 2026-10-30. This list used to name only write:repository + read:user (+ read:package),
# and the operator minted exactly that. Run 350 then failed at
# `GET /api/v1/orgs/Home-Network 403 ... required=[read:organization], token scope=read:package,
# write:repository,read:user`; write:issue would have been the next wall. The preflight below now
# PROBES these scopes, so a short token fails in seconds with the missing scope named.
# Set it as an Actions secret on this repository. Nothing in this repo mints,
# stores or references its value; it is consumed by name only.
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
# visible red is the correct state for "automation configured but not credentialed".
@@ -74,12 +87,26 @@ name: renovate
on:
schedule:
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
# of the repo window means a run is always in flight when the window opens.
- cron: "15 3 * * *"
# 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
# `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
# it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
# merely near it.
#
# WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
# run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
# 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
# Every nightly run would have started, logged "not within schedule", opened zero PRs and
# exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
# green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
# needed updating".
#
# THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
# EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
# EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
# 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
# schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
# early-morning-UTC value without redoing this arithmetic.
- cron: "15 8 * * *"
workflow_dispatch:
inputs:
dry_run:
@@ -101,15 +128,37 @@ concurrency:
jobs:
renovate:
runs-on: ubuntu-latest
# RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
# That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
# this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
# act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
# is privileged, so a job that could reach it would be root on the node). So every run of the
# action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
# [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
# failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
# ::error::Error: The process '/usr/bin/docker' failed with exit code 1
# No auth, scope, config or network question was ever reached. Making the Renovate image the
# job container needs no daemon: act starts it through the runner's own DinD exactly as it
# starts the default ubuntu image, then `exec`s each step inside it.
#
# PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
# against this exact tag — keep them in step. Renovate's github-actions manager reads
# `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
#
# The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
# ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
# HOME comes from the image env, and no step writes the workspace. There is deliberately NO
# actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
# checkout could not `git init` the root-owned workspace volume anyway.
container:
image: ghcr.io/renovatebot/renovate:44.167.0
# Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
# well under a runner's patience. Without a timeout a wedged run holds the `renovate`
# concurrency group and silently blocks every subsequent scheduled run.
timeout-minutes: 40
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Preflight - the token must exist, and its absence must be LOUD
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
run: |
@@ -124,23 +173,94 @@ jobs:
That is the failure mode that let renovate.json sit inert in this repo for months while
seven files claimed "Renovate maintains this pin".
TO FIX: create a Gitea access token for the bot account with scopes
`write:repository` and `read:user`, then add it to this repository as the Actions
secret RENOVATE_TOKEN. Do not commit it anywhere.
TO FIX: create a Gitea access token for the bot account (renovate-bot) with scopes
read:user, read:organization, write:repository, write:issue and read:package (the
header of .gitea/workflows/renovate.yaml says which call needs which), then add it to
this repository as the Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
MSG
exit 1
fi
echo "RENOVATE_TOKEN is present (value not printed)."
# Proves the step is running INSIDE the pinned Renovate image, not the default job image.
echo "renovate $(renovate --version)"
- name: Preflight - the token has every scope Renovate needs
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
# ADDED 2026-10-30. Renovate reports EVERY platform-init failure as the bare
# "Authentication failure" (lib/modules/platform/gitea/index.ts initPlatform swallows the
# HTTP error), and a later scope gap surfaces as "Repository has unknown error". Two
# dispatches were spent learning one missing scope at a time. These probes are GETs only,
# and they name the missing scope. The token never reaches argv: curl reads it from stdin
# (`--config -`), so it is not in `ps` output or in the log.
run: |
set -uo pipefail
GITEA=https://gitea.home.keithjackson.dev
REPO=Home-Network/home-network-k3s-impl
fail=0
body=$(mktemp)
probe() { # <scope it proves> <path> <auth: bearer|basic>
local code
if [ "$3" = basic ]; then
code=$(printf 'user = "renovate:%s"\n' "$RENOVATE_TOKEN" \
| curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
else
code=$(printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
| curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
fi
if [ "$code" = 200 ]; then
echo "ok $1 (GET ${2%%\?*} -> 200)"
return
fi
fail=1
echo "::error::scope $1: GET ${2%%\?*} -> $code"
case "$code" in
401) echo " 401 = Gitea rejected the token VALUE (wrong, expired or revoked)." ;;
403) if grep -q 'Access denied' "$body"; then
echo " 403 'Access denied' = a Cilium/Envoy policy denial, not Gitea."
else
echo " Gitea: $(jq -r '.message // empty' "$body" 2>/dev/null | head -c 300)"
fi ;;
*) echo " body: $(head -c 200 "$body")" ;;
esac
}
probe read:user "/api/v1/user" bearer
probe read:organization "/api/v1/orgs/${REPO%%/*}" bearer
probe read:repository "/api/v1/repos/$REPO" bearer
probe read:issue "/api/v1/repos/$REPO/labels?limit=1" bearer
probe read:package "/v2/token?service=container_registry&scope=repository:home-network/toolbox:pull" basic
# WRITE scopes cannot be proven by a GET. Gitea's scope-refusal message, however, lists
# the token's scopes (names only, never the value): ask for an admin route this bot
# cannot have, and read `token scope=` from the 403. Advisory if the format changes.
printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
| curl -sS --config - -o "$body" "$GITEA/api/v1/admin/orgs?limit=1" || true
scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
if [ -n "$scopes" ]; then
echo "token scopes (from Gitea): $scopes"
for want in write:repository write:issue; do
case ",$scopes," in
*",$want,"*|*",all,"*) echo "ok $want" ;;
*) echo "::error::scope $want is missing (token scope=$scopes)"; fail=1 ;;
esac
done
else
echo "::warning::could not read the token's scope list; write:repository and write:issue are UNVERIFIED"
fi
rm -f "$body"
if [ "$fail" -ne 0 ]; then
echo "::error::RENOVATE_TOKEN failed the checks above (a 401 is the value, a 403 is a scope). Re-mint it for renovate-bot with read:user, read:organization, write:repository, write:issue, read:package and overwrite the secret."
exit 1
fi
- name: Run Renovate
uses: renovatebot/[email protected]
with:
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
# This is the one dependency in the repo whose updater is its own updater.
configurationFile: renovate.json
token: ${{ secrets.RENOVATE_TOKEN }}
shell: bash
# The repo config is discovered by Renovate from the cloned repo (renovate.json5,
# RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
# via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
# check therefore has nothing to resolve in this file, and passes.
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
# header said were "set at runtime by the operator's renovate runner" — this is that
# runner, so they are set here rather than in the repo config, which keeps renovate.json
@@ -156,6 +276,31 @@ jobs:
RENOVATE_REQUIRE_CONFIG: "required"
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
# ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
# agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
# followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
# anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
# docker lookup never matches — so without this rule every digest lookup fails. Same
# token; it therefore needs scope `read:package` in addition to the ones listed in the
# preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
# OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
# public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
# SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
# every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
# and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
# empty variable, and those deps simply stay unmaintained, with a WARN in the log.
GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
run: |
set -euo pipefail
# An expression that evaluates to '' still EXPORTS the variable, as an empty string.
# Drop those so an unset value means "Renovate's default", never "set to empty".
for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
[ -n "${!v:-}" ] || unset "$v"
done
# Through the image's entrypoint, as `docker run` would have gone: it sources the
# containerbase env and initialises the bundled tools before exec'ing Renovate.
exec renovate-entrypoint.sh renovate
- name: Summary
if: always()
+373
View File
@@ -0,0 +1,373 @@
#!/usr/bin/env bash
###############################################################################
# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
#
# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
#
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
#
# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
#
# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
# builds `<image>-verify` and only then the image. Build args come from each image's own
# `build-image.sh --print-build-args`, the same list the local podman path uses.
#
# MODES (env MODE):
# verify build the verify targets, never push, hold no registry credential. Pull requests.
# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
# it). The digest recorded is the one the REGISTRY returns, read back after the push.
#
# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
#
# THE agent-runner BASE:
# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
# a pull request holds no registry credential and that toolbox exists in no registry. If the
# toolbox's inputs did not change it is built anyway, as the base.
# * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
# toolbox has never been published, so it is built and published first (the bootstrap).
#
# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
###############################################################################
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
cd "${REPO_ROOT}"
# shellcheck disable=SC1091
BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
. ./platform/buildkit/versions.env
BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
MODE="${MODE:-verify}"
IMAGES="${IMAGES:-auto}"
REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
NAMESPACE="${NAMESPACE:-home-network}"
HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
BASE_SHA="${BASE_SHA:-}"
SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
ALL_IMAGES=(toolbox agent-runner conformance-runner)
SHA_TAG="sha-${HEAD_SHA:0:12}"
MOVING_TAG="main"
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
# --- per-image facts ------------------------------------------------------------------------
# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
# published digest), and must not become one: Renovate writes it after every publish -> a loop.
inputs() {
case "$1" in
toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
agent-runner) echo "infra/agent/" ;;
conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
esac
}
context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
[ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
}
select_images() {
local i
if [ "${IMAGES}" != "auto" ]; then
for i in ${IMAGES}; do
case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
done
return
fi
if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
info "no usable diff base ('${BASE_SHA}'): building everything"
for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
return
fi
info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
for i in "${ALL_IMAGES[@]}"; do
# shellcheck disable=SC2046
if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
done
# shellcheck disable=SC2086
if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
fi
}
# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
WORK="$(mktemp -d)"
trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long
export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
# about one manifest per tag. (Provenance attestations would turn every push into an index.)
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
echo '{}' > "${DOCKER_CONFIG}/config.json"
setup_builder() {
log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
BUILDX="${BUILDX_BIN:-}"
if [ -z "${BUILDX}" ]; then
BUILDX="${WORK}/buildx-bin"
curl -fsSL --retry 3 -o "${BUILDX}" \
"https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
|| die "could not download buildx v${BUILDX_VERSION}"
echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \
|| die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
chmod +x "${BUILDX}"
fi
info "client: $("${BUILDX}" version)"
local missing=()
[ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA)
[ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
[ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY)
[ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
umask 077
printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem"
printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem"
umask 022
"${BUILDX}" create --name ci-remote --driver remote \
--driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
"${BUILDKIT_HOST}" >/dev/null
if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
printf '%s\n' "${out}" >&2
die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
rather than skipping. In order of likelihood:
1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
(Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
re-run infra/gitea/buildkit-client-secret.sh.
3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
fi
printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true
}
# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
build_args_json() {
local ctx; ctx="$(context_of "$1")"
"./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
}
# bake_run <bakefile> <targets...>
bake_run() {
local f="$1"; shift
"${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
--metadata-file "${WORK}/meta.json" "$@"
}
# Write a bake file with a verify target and a ship target for one image.
# $1 image, $2 agent base mode: "" | "link" | <image ref>
write_bake() {
local img="$1" base="${2:-}" ctx args f
ctx="$(context_of "$img")"
args="$(build_args_json "$img")"
f="${WORK}/bake-${img}.json"
local contexts='{}'
if [ "${img}" = agent-runner ]; then
if [ "${base}" = link ]; then
# FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
# of this same bake file — built here, never pushed anywhere.
args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
contexts='{"toolbox-in-this-run": "target:toolbox"}'
else
args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
fi
fi
jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
--arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
--arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
--arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
{target: (
{
($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
args: $args, contexts: $contexts, output: ["type=cacheonly"]},
($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
contexts: $contexts,
labels: {"org.opencontainers.image.revision": $rev}}
+ (if $mode == "publish"
then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
else {output: ["type=cacheonly"]} end))
}
+ (if ($contexts | has("toolbox-in-this-run"))
then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
else {} end)
)}' > "$f"
echo "$f"
}
# --- registry helpers (publish mode only) ----------------------------------------------------
REG_TOKEN=""
registry_token() {
[ -n "${REG_TOKEN}" ] && return 0
REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
"https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
|| die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
[ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
}
# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
# immutable-tag guard would be bypassed by an auth hiccup.
registry_digest() {
registry_token
local hdrs code digest
hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
-H 'Accept: application/vnd.oci.image.index.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
"https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
|| die "registry unreachable while reading $1:$2"
code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
case "${code}" in
200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
echo "${digest}" ;;
404) ;;
*) die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
esac
}
# build_one <image> [agent base] — verify, then (publish) build + push + read back.
build_one() {
local img="$1" base="${2:-}" t0 f pushed got existing
t0=$(date +%s)
f="$(write_bake "$img" "$base")"
log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
bake_run "$f" "${img}-verify"
BUILT[$img]=1
if [ "${MODE}" = publish ]; then
existing="$(registry_digest "${img}" "${SHA_TAG}")"
# Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
# silently change what `sha-<commit>` means after something may already have pinned it.
[ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
bake_run "$f" "${img}"
pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
# reported. They should agree; if they do not, nothing downstream may trust either.
got="$(registry_digest "${img}" "${SHA_TAG}")"
[ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
DIGEST[$img]="${got}"
info "${img}: registry digest ${got}"
fi
SECS[$img]=$(( $(date +%s) - t0 ))
info "${img}: done in ${SECS[$img]}s"
}
# --- main ---------------------------------------------------------------------------------
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
select_images
if [ "${MODE}" = publish ]; then
: "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
# buildx (and through it buildkitd, which does the push) reads registry auth from this file.
jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
'{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
chmod 600 "${DOCKER_CONFIG}/config.json"
registry_token
# Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
# makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
for i in "${ALL_IMAGES[@]}"; do
[ -n "${WANT[$i]:-}" ] || continue
d="$(registry_digest "${i}" "${SHA_TAG}")"
if [ -n "${d}" ]; then
info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
fi
done
fi
# Decide the agent-runner base (see header).
AGENT_BASE=""
if [ -n "${WANT[agent-runner]:-}" ]; then
PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
if [ "${MODE}" = verify ]; then
AGENT_BASE=link
[ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
case "${PINNED_BASE}" in
*@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
*@sha256:*) AGENT_BASE="${PINNED_BASE}" ;;
*) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
esac
fi
fi
for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done
if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
info "nothing to build"
else
setup_builder
fi
if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
if [ -n "${WANT[agent-runner]:-}" ]; then
if [ -z "${AGENT_BASE}" ]; then
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
fi
info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
build_one agent-runner "${AGENT_BASE}"
fi
if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
# --- summary ------------------------------------------------------------------------------
{
echo "## Images (${MODE})"
echo
echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
echo
echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
echo "|---|---|---|---|---|"
for i in "${ALL_IMAGES[@]}"; do
d="${DIGEST[$i]:-}"
echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
done
echo
if [ "${MODE}" = publish ]; then
echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
else
echo "Verify mode: nothing was pushed and no registry credential was present."
fi
} >>"${SUMMARY}"
log "done"
+1 -1
View File
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
+5
View File
@@ -100,6 +100,11 @@ metadata:
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
+2 -2
View File
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover.
+7 -3
View File
@@ -78,7 +78,7 @@ spec:
spec:
containers:
- name: gitea
image: gitea/gitea:1.24.3
image: gitea/gitea:28.3.0
ports:
- containerPort: 3000
env:
@@ -89,14 +89,18 @@ spec:
secretKeyRef:
name: gitea-bootstrap
key: secret-key
- name: GITEA__server__DOMAIN
value: gitea.${BASE_DOMAIN}
- name: GITEA__server__ROOT_URL
value: https://gitea.${BASE_DOMAIN}/
- name: GITEA__actions__ENABLED
value: 'true'
- name: GITEA__actions__DEFAULT_ACTIONS_URL
value: github
- name: GITEA__actions__RUN_RETENTION_DAYS
value: '400'
- name: GITEA__security__EGRESS_MODE
value: lax
- name: GITEA__service__DISABLE_REGISTRATION
value: 'true'
- name: GITEA_ADMIN_USERNAME
valueFrom:
secretKeyRef:
+2 -2
View File
@@ -78,14 +78,14 @@ spec:
spec:
containers:
- name: jackett
image: lscr.io/linuxserver/jackett:latest
image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
ports:
- containerPort: 9117
volumeMounts:
- name: config
mountPath: /config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
+2 -2
View File
@@ -48,7 +48,7 @@ spec:
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
imageName: ghcr.io/cloudnative-pg/postgresql:16.15
# Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
# nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
@@ -300,7 +300,7 @@ spec:
# is already in it and it is already present on every node, so this
# adds no new image and no new supply-chain surface. Keep the tag in
# step with spec.imageName above when Renovate bumps it.
image: ghcr.io/cloudnative-pg/postgresql:16.4
image: ghcr.io/cloudnative-pg/postgresql:16.15
imagePullPolicy: IfNotPresent
command:
- psql
+19 -5
View File
@@ -83,15 +83,25 @@
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
#
# -------------------------------------------------------------------------------------------------
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
# -------------------------------------------------------------------------------------------------
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
# range to a /24 that excludes the test client, after which the same query that had been answering
# stopped being answered at all, and restoring it, after which it answered again.
#
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
# if either entry goes missing or any range here stops being RFC1918.
#
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
# arrives from the ingress node's internal router address and Cilium resolves that source to the
@@ -113,6 +123,7 @@ spec:
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
@@ -133,6 +144,7 @@ spec:
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
@@ -158,7 +170,9 @@ spec:
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
#
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
@@ -187,7 +201,7 @@ spec:
app.kubernetes.io/name: pihole
ingress:
- fromEntities:
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
- world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
- host # this node's kubelet probes
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
toPorts:
+3 -3
View File
@@ -62,7 +62,7 @@ spec:
spec:
containers:
- name: pihole
image: pihole/pihole
image: pihole/pihole:2026.10.1
ports:
- containerPort: 80
env:
@@ -73,7 +73,7 @@ spec:
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password
valueFrom:
secretKeyRef:
@@ -90,7 +90,7 @@ spec:
- -c
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
+3 -1
View File
@@ -120,7 +120,7 @@ spec:
spec:
containers:
- name: plex
image: plexinc/pms-docker:latest
image: plexinc/pms-docker:1.43.6.11085-d1e52e169
ports:
- containerPort: 32400
env:
@@ -302,6 +302,8 @@ metadata:
namespace: plex
labels:
app.kubernetes.io/name: plex
annotations:
external-dns.alpha.kubernetes.io/controller: none
spec:
parentRefs:
- name: shared
+2 -2
View File
@@ -122,7 +122,7 @@ spec:
spec:
containers:
- name: qbittorrent
image: lscr.io/linuxserver/qbittorrent:5.1.2
image: lscr.io/linuxserver/qbittorrent:5.3.0
ports:
- containerPort: 8080
env:
@@ -156,7 +156,7 @@ spec:
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
+1 -1
View File
@@ -49,7 +49,7 @@ spec:
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
imageName: ghcr.io/cloudnative-pg/postgresql:16.15
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
+71 -7
View File
@@ -77,6 +77,42 @@ spec:
requests:
storage: 1Gi
---
# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
# unset field with the same default it uses when there is no file at all, so a file holding
# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
# no-file default (false); and its sample `envs` would be injected into every job.
#
# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
# below is consumed by the image's run.sh for registration, not by that legacy path.
#
# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
# config-revision annotation below) or the new value stays inert until the next restart.
apiVersion: v1
kind: ConfigMap
metadata:
name: act-runner-config
namespace: gitea-actions
labels:
app.kubernetes.io/name: act-runner
data:
config.yaml: |
runner:
# Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
# 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
# renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
# containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
# collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
# unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
# and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
capacity: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -95,6 +131,10 @@ spec:
metadata:
labels:
app.kubernetes.io/name: act-runner
annotations:
# Bump when act-runner-config changes without any other pod-template change, so the
# Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
homelab.keithjackson.dev/act-runner-config-revision: "1" # capacity 2
spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14).
@@ -133,7 +173,7 @@ spec:
initContainers:
- name: dind
restartPolicy: Always # <- this field is what makes it a SIDECAR
image: docker:27-dind
image: docker:29-dind
securityContext:
privileged: true
env:
@@ -146,12 +186,23 @@ spec:
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
# Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
# container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
# capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
# cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
# parallelised, so per-job demand rises. The request is what the CFS share is
# weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
# so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
# requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
# memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
# takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
# over that for the parallelised gate. Nodes have ~31Gi allocatable.
resources:
requests:
cpu: 100m
memory: 256Mi
cpu: "2"
memory: 1Gi
limits:
memory: 2Gi
memory: 4Gi
startupProbe:
exec:
command:
@@ -167,7 +218,7 @@ spec:
containers:
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
- name: runner
image: gitea/act_runner:0.2.11
image: gitea/act_runner:0.6.1
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
# that used to live here are gone: kubelet now holds this container until the dind
@@ -195,23 +246,36 @@ spec:
value: /certs/client
- name: DOCKER_TLS_VERIFY
value: "1"
# The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
# `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
- name: CONFIG_FILE
value: /etc/act_runner/config.yaml
volumeMounts:
- name: data
mountPath: /data
- name: certs
mountPath: /certs
readOnly: true
- name: config
mountPath: /etc/act_runner
readOnly: true
# Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
# against the old 512Mi limit. It holds per-task state, log buffers and the embedded
# actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
resources:
requests:
cpu: 100m
memory: 128Mi
memory: 256Mi
limits:
memory: 512Mi
memory: 1Gi
volumes:
- name: data
persistentVolumeClaim:
claimName: act-runner-data
- name: certs
emptyDir: {}
- name: config
configMap:
name: act-runner-config
- name: dind-storage
emptyDir: {}
+1 -1
View File
@@ -75,7 +75,7 @@ spec:
spec:
containers:
- name: keycloak
image: quay.io/keycloak/keycloak
image: quay.io/keycloak/keycloak:26.0.6
ports:
- containerPort: 8080
volumeMounts:
+2 -2
View File
@@ -122,7 +122,7 @@ spec:
spec:
containers:
- name: radarr
image: lscr.io/linuxserver/radarr:5.21.1
image: lscr.io/linuxserver/radarr:5.28.0
ports:
- containerPort: 7878
env:
@@ -138,7 +138,7 @@ spec:
- name: media
mountPath: /media
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
+1 -1
View File
@@ -78,7 +78,7 @@ spec:
spec:
containers:
- name: semaphore
image: semaphoreui/semaphore:v2.19.10
image: semaphoreui/semaphore:v2.19.17
ports:
- containerPort: 3000
env:
+2 -2
View File
@@ -122,7 +122,7 @@ spec:
spec:
containers:
- name: sonarr
image: lscr.io/linuxserver/sonarr:4.0.15
image: lscr.io/linuxserver/sonarr:4.0.20
ports:
- containerPort: 8989
env:
@@ -138,7 +138,7 @@ spec:
- name: media
mountPath: /media
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
+1 -1
View File
@@ -75,7 +75,7 @@ spec:
spec:
containers:
- name: seerr
image: ghcr.io/seerr-team/seerr:v3.4.1
image: ghcr.io/seerr-team/seerr:v3.5.1
ports:
- containerPort: 5055
volumeMounts:
+39 -5
View File
@@ -10,7 +10,7 @@
name: gitea
kind: container
source:
image: gitea/gitea:1.24.3 # pinned recent stable tag (R-UPD-01); update.range governs advancement
image: gitea/gitea:28.3.0 # pinned recent stable tag (R-UPD-01); update.range governs advancement
# Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
# ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
# without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
@@ -23,14 +23,43 @@ env:
secretKeyRef:
name: gitea-bootstrap
key: secret-key
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
value: gitea.${BASE_DOMAIN}
# ROOT_URL is the ONLY source of the instance domain since Gitea 28 (go-gitea/gitea#39400): [server]
# DOMAIN is no longer read, and the SSH domain defaults from ROOT_URL too. GITEA__server__DOMAIN was
# removed in the 28 upgrade; the stale DOMAIN line environment-to-ini already wrote into app.ini is
# inert. Correct public clone/redirect URLs for the TLS-terminated host come from this one value.
- name: GITEA__server__ROOT_URL
value: https://gitea.${BASE_DOMAIN}/
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
value: github # has internet egress; not air-gapped)
# Gitea 28 added run retention (go-gitea/gitea#38855): completed runs, with their jobs, logs and
# artifacts, are deleted by the cleanup_action_runs cron after RUN_RETENTION_DAYS. Upstream's default
# (400) is kept but DECLARED, so the deletion horizon is visible here rather than implicit. The spec
# repo states no CI-history retention requirement; 0 would mean keep forever.
- name: GITEA__actions__RUN_RETENTION_DAYS
value: "400"
# Gitea 28 (go-gitea/gitea#39426) puts webhooks, the OAuth2 avatar fetch and OpenID 2.0 behind an
# egress policy: in the default lax mode, PUBLIC targets are allowed and private/loopback/CGNAT
# targets need an IP or CIDR entry in [security] ALLOWED_HOST_LIST (a hostname entry is NOT enough
# for a private address). DECLARED lax with an EMPTY allow list, deliberately:
# * the Keycloak OIDC login source is NOT gated by it. In 28.3.0 the openidConnect provider is
# goth's, built with its default HTTP client (services/auth/source/oauth2/providers_openid.go);
# egress.NewSecurityPolicy is used only by routers/web/auth/oauth.go (avatar fetch, and
# [oauth2_client] UPDATE_AVATAR is off here) and modules/auth/openid (OpenID 2.0, unused).
# So keycloak.<domain> resolving to the private Gateway address in-cluster does not matter.
# * this instance has no webhooks and no mirrors (measured 2026-11-07), and 1.27 already refused
# private webhook targets by default, so nothing that worked before is newly denied.
# Adding an entry here would only WIDEN the webhook SSRF surface. Add the narrowest CIDR (e.g. the
# Gateway address /32) only if a real private webhook target appears.
- name: GITEA__security__EGRESS_MODE
value: lax
# Gitea 28 (go-gitea/gitea#39400) disables LOCAL self-registration unless DISABLE_REGISTRATION is
# explicitly false; 1.27 defaulted it open. Declared true: accounts arrive via Keycloak only.
# OAuth2 auto-registration below is unaffected — routers/web/auth/oauth.go gates it on
# ALLOW_ONLY_INTERNAL_REGISTRATION and ENABLE_AUTO_REGISTRATION, not on DISABLE_REGISTRATION.
- name: GITEA__service__DISABLE_REGISTRATION
value: "true"
- name: GITEA_ADMIN_USERNAME # break-glass local admin creds (consumed by the postStart hook)
valueFrom:
secretKeyRef:
@@ -233,9 +262,14 @@ auth:
# recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
update:
channel: stable # R-UPD-01
range: ">=1.22.0 <2.0.0" # stay on the 1.x stable line
range: ">=28.0.0 <29.0.0" # Gitea dropped the "1." prefix at 28.0.0 (successor of 1.27); stay on 28.x
cadence: "0 4 * * *"
autoRollbackOnFailedHealth: true # R-UPD-03
# R-UPD-03. INERT FOR THIS SERVICE, and it must stay that way: this flag renders only an annotation
# on a Flux ImagePolicy, and apps/ is rendered --no-image-automation (ci/render-golden.sh), so no
# ImagePolicy exists and the Deployment (strategy Recreate) is never auto-reverted. That matters
# because a Gitea upgrade (1.27 -> 28 included) MIGRATES THE SQLITE DB ONE-WAY: an image revert onto
# a migrated DB is not a rollback. Roll Gitea back by RESTORING THE DATA PVC from its pre-upgrade backup.
autoRollbackOnFailedHealth: true
observability:
metrics: true
slis:
+3 -2
View File
@@ -16,7 +16,7 @@
name: jackett
kind: container
source:
image: lscr.io/linuxserver/jackett:latest
image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
storage:
@@ -70,7 +70,8 @@ auth:
- ^/api/
update:
channel: stable # R-UPD-01
range: ">=0.22.0" # stay on the Jackett 0.22.x stable line
range: ">=0.22.0" # floor only, no ceiling (0.24.x runs as of 2026-10-30; Jackett has no
# major line to hold back from, so renovate.json5 mirrors no cap)
cadence: "0 4 * * *"
autoRollbackOnFailedHealth: true # R-UPD-03
observability:
+8 -1
View File
@@ -5,8 +5,15 @@
# Validates against schemas/servicedefinition.schema.json.
name: keycloak
kind: container
# DOCUMENTATION-ONLY IMAGE. The running server is NOT chosen here: the Keycloak operator
# (platform/keycloak/kustomization.yaml, three pinned release URLs) picks it via its
# RELATED_IMAGE_KEYCLOAK, and the Keycloak CR has no spec.image. This SD's render,
# apps/wave-1/keycloak.yaml, is not listed in apps/wave-1/kustomization.yaml and deploys nothing
# (ci/render-golden.sh RENDER_GOLDEN_NOT_ONBOARDED). The tag records the operator's version for
# readers; renovate.json5 disables Renovate for this image so it cannot raise PRs against a file
# nothing deploys. The Keycloak version moves with the operator — update this tag by hand then.
source:
image: quay.io/keycloak/keycloak
image: quay.io/keycloak/keycloak:26.0.6
exposure: public # login/redirect flows for internet-exposed services; hardened (docs/15; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
+89 -31
View File
@@ -5,7 +5,7 @@
name: pihole
kind: container
source:
image: pihole/pihole
image: pihole/pihole:2026.10.1
exposure: internal # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
@@ -85,7 +85,7 @@ env:
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
- name: FTLCONF_dns_listeningMode
@@ -108,43 +108,101 @@ env:
# `loadBalancer.mode: dsr`, which preserves the client address.
- name: FTLCONF_dns_rateLimit_count
value: "0"
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
# so the first line is a wildcard for the whole zone and each later line is a specific override.
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
# changes nothing.
# THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
# `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
# boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
# LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
# FORWARDED to the upstreams above and gets the PUBLIC answer.
#
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
# is the annotation that means "this Service publishes its own hostname at its own address":
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
# resolver address, are the complete set of exceptions today.
# WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
# ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
# DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
# private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
# (platform/externaldns/README.md), which works identically from the LAN — so a name that is
# public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
# it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
# the same working answer. The wildcard had also become actively misleading: it answered names
# that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
#
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
# WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
# -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
#
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
# now get Cloudflare's real A and AAAA answers.
#
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
# while resolution for the zone does not work. Confirmed both ways against the live image.
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
# every client that takes the ADVERTISED connection cannot reach the server at all.
#
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
# (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
# (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
# Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
# 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
# reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
# before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
# 32400; without it now, the name is NXDOMAIN.
#
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
#
# dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
# dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
# dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
#
# dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
# bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
# * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
# single lookup, which is maximum exposure to any momentary gap; while
# * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
# upstream (standard forwarder behaviour when the forward target does not answer) it latches
# the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
# The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
# live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
# flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
# That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
#
# 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
# re-election (both seconds), short enough that a genuine VIP change still propagates inside a
# minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
# propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
# LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
# the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
# cache expires, the router falls back, and the public answer latches again. That is a real
# outage, and the fix for it is PiHole availability, not DNS caching.
#
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
#
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
# configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
- name: FTLCONF_misc_dnsmasq_lines
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
# carries the full rationale and the measurement that found it.
+9 -2
View File
@@ -12,7 +12,7 @@
name: plex
kind: container
source:
image: plexinc/pms-docker:latest
image: plexinc/pms-docker:1.43.6.11085-d1e52e169
# plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
env:
# PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
@@ -347,7 +347,14 @@ lifecycle:
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
hardware:
gpu: intel-quicksync
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
exposure: internal # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
- name: config
+1 -1
View File
@@ -30,7 +30,7 @@
name: qbittorrent
kind: container
source:
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
image: lscr.io/linuxserver/qbittorrent:5.3.0 # pinned stable tag (R-UPD-01); update.range governs advancement
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
+1 -1
View File
@@ -8,7 +8,7 @@
name: radarr
kind: container
source:
image: lscr.io/linuxserver/radarr:5.21.1 # pinned stable v5 tag (R-UPD-01); update.range governs advancement
image: lscr.io/linuxserver/radarr:5.28.0 # pinned stable v5 tag (R-UPD-01); update.range governs advancement
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
# which cannot write a 1000:3000-owned 0755 tree.
+1 -1
View File
@@ -129,7 +129,7 @@ AUTH_MODES = ("oidc", "forward-auth", "native", "none")
# the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups)
# and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN
# the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02).
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.15.6"
OAUTH2_PROXY_PORT = 4180
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
+1 -1
View File
@@ -24,7 +24,7 @@
name: seerr
kind: container
source:
image: ghcr.io/seerr-team/seerr:v3.4.1 # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
image: ghcr.io/seerr-team/seerr:v3.5.1 # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
# The ghcr.io/seerr-team/seerr image runs as the non-root user `node:node` (UID 1000 / GID 1000 —
# the standard node base-image user; confirmed from the image config's User field). On boot the app
# creates /app/config/logs and writes its settings/DB under the mounted /app/config PVC. A fresh
+1 -1
View File
@@ -22,7 +22,7 @@
name: semaphore
kind: container
source:
image: semaphoreui/semaphore:v2.19.10 # pinned recent stable tag (R-UPD-01); update.range governs advancement
image: semaphoreui/semaphore:v2.19.17 # pinned recent stable tag (R-UPD-01); update.range governs advancement
env:
# --- PostgreSQL connection (the CNPG-DB pattern). DIALECT/HOST/PORT/DB/USER are literals; PASS is
# sourced from the in-namespace SOPS secret (the CNPG -app secret lives in ns semaphore-db,
+1 -1
View File
@@ -8,7 +8,7 @@
name: sonarr
kind: container
source:
image: lscr.io/linuxserver/sonarr:4.0.15 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
image: lscr.io/linuxserver/sonarr:4.0.20 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
# which cannot write a 1000:3000-owned 0755 tree.
+1 -1
View File
@@ -99,7 +99,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
| stage | a skip there means | verdict |
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is the job image — the CI-built toolbox, `.gitea/workflows/gate.yml` `container:` — so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
+3 -3
View File
@@ -46,8 +46,8 @@
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
#
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
# seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
# CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
@@ -257,7 +257,7 @@ for s in lint l0 render-golden l1; do
skip)
if strict_skips; then
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
err " toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
BLOCK=1
+3 -3
View File
@@ -97,9 +97,9 @@ cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
#
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
# before the gate runs, and every seam is on `main`. A stage that skips here is a
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
# CI the pinned toolchain IS the job image (the CI-built toolbox; gate.yml `container:`),
# and every seam is on `main`. A stage that skips here is a
# DEFECT (a tool dropped from the image, a seam that went unwired) -> BLOCK.
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
#
+69 -2
View File
@@ -25,5 +25,72 @@ if [ "${#files[@]}" -eq 0 ]; then
exit 0
fi
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
ok "yaml/lint passed (${#files[@]} files)"
# yamllint is single-threaded, so the file list is split into LINT_JOBS CONTIGUOUS slices of the
# sorted list, one yamllint per slice, run concurrently. Each slice's output goes to its own file
# and the files are printed in slice order afterwards: the log is the same text, in the same
# file order, as one serial `yamllint "${files[@]}"`, and a finding is never interleaved with
# another process's output (yamllint's standard format puts the FILE NAME on its own line above
# that file's `line:col` findings, so interleaving would misattribute them). The stage's exit
# code is the highest any slice returned (yamllint: 1 = error, 2 = warning under strict), so a
# single failing slice fails the stage exactly as the serial call did. LINT_JOBS=1 is serial.
lint_jobs="${LINT_JOBS:-$(n="$(nproc 2>/dev/null || echo 1)"; [ "$n" -gt 8 ] && n=8; echo "$n")}"
case "$lint_jobs" in ''|*[!0-9]*|0) err "LINT_JOBS must be a positive integer, got '${lint_jobs}'"; exit 2 ;; esac
[ "$lint_jobs" -gt "${#files[@]}" ] && lint_jobs="${#files[@]}"
lint_tmp="$(mktemp -d)"
trap 'rm -rf "$lint_tmp"' EXIT
per=$(( (${#files[@]} + lint_jobs - 1) / lint_jobs ))
pids=()
for (( i = 0; i < lint_jobs; i++ )); do
slice=("${files[@]:$(( i * per )):${per}}")
[ "${#slice[@]}" -gt 0 ] || break
( rc=0
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${slice[@]}" \
>"${lint_tmp}/${i}.out" 2>&1 || rc=$?
echo "$rc" >"${lint_tmp}/${i}.rc" ) &
pids+=("$!")
done
wait "${pids[@]}" || true
lint_rc=0
for (( i = 0; i < ${#pids[@]}; i++ )); do
cat "${lint_tmp}/${i}.out"
# A slice that left no rc file died before yamllint returned: that is a failure, not a pass.
slice_rc="$(cat "${lint_tmp}/${i}.rc" 2>/dev/null || true)"
case "$slice_rc" in ''|*[!0-9]*) slice_rc=1 ;; esac
[ "$slice_rc" -gt "$lint_rc" ] && lint_rc="$slice_rc"
done
if [ "$lint_rc" -ne 0 ]; then
err "yaml/lint failed (yamllint exit ${lint_rc}; ${#files[@]} files in ${#pids[@]} parallel slice(s))"
exit "$lint_rc"
fi
ok "yaml/lint passed (${#files[@]} files, ${#pids[@]} parallel slice(s))"
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
# while the gate stayed green. Valid YAML, so yamllint cannot see it.
missing=0
while IFS=: read -r wf _ value; do
value="${value#*configurationFile:}"
value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
if [ -n "$value" ] && [ ! -e "$value" ]; then
err "${wf}: configurationFile '${value}' does not exist in the repo"
missing=1
fi
done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
[ "$missing" -eq 0 ] || exit 1
ok "workflow configurationFile references resolve"
# A workflow must not use an action that needs a Docker daemon: jobs on this runner get NONE, by
# decision (decisions/2026-10-29-image-pipeline.md; act_runner maps docker_host to `-`). Such an
# action is valid YAML and passes every other stage, then dies at runtime with
# "failed to connect to the docker API at unix:///var/run/docker.sock". renovatebot/github-action
# did exactly that on every Renovate run until 2026-10-30 (run 343). Run the tool's image as the
# job `container:` instead (see .gitea/workflows/renovate.yaml). Container (`docker://`) actions
# and Docker's own daemon-bound actions are rejected for the same reason. Comments are ignored.
daemon_uses='uses:[[:space:]]*["'\'']?(docker://|renovatebot/github-action|docker/(build-push|setup-buildx|setup-qemu|login)-action)'
bad=0
while IFS= read -r hit; do
err "${hit%%:*}: needs a Docker daemon, which jobs here do not get: $(echo "${hit#*:*:}" | sed -E 's/^[[:space:]]+//')"
bad=1
done < <(grep -HnE "^[^#]*${daemon_uses}" .gitea/workflows/*.y*ml 2>/dev/null || true)
[ "$bad" -eq 0 ] || exit 1
ok "no workflow uses a Docker-daemon-bound action"
+490
View File
@@ -0,0 +1,490 @@
#!/usr/bin/env python3
# =============================================================================
# ci/render-keycloak-passkey-policy.py
#
# Render platform/keycloak/passkey-desired-state.yaml FROM
# platform/keycloak/keycloak-realm-import.yaml.
#
# WHY THIS EXISTS
# ---------------
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
# after the passkey declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
# Done=True Started=False HasErrors=False
#
# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
# is the level-triggered loop that closes that; this renderer supplies the DESIRED
# STATE it converges towards.
#
# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
# flow in a browser; make it the login path" -- not a fact derivable from the realm
# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
# file: --write preserves it, --check accepts any value of it that is exactly the
# string "true" or "false" and rejects everything else (an unquoted YAML true is a
# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
# Kustomization -- so it is refused HERE, at merge time). An absent file or key
# renders as "false". Going live is therefore a one-line PR flipping that line.
#
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
# automountServiceAccountToken: false, exactly like its client-secret sibling. It
# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
# must be MOUNTED. Hence a ConfigMap.
#
# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
# the user-verification requirement and the whole flow tree — free to disagree with
# the realm import. "Two declarations that can disagree" is the same shape as the
# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
# import and the generation is GATED:
# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
# with --check on every merge, and the merge fails if the committed file is not what
# this renderer produces. Same render==golden idiom as
# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
#
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
# rendered by the SAME Flux Kustomization (./platform-config, whose
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
# pass with the same value and cannot disagree about what the base domain is. This
# renderer never parses, splits or reconstructs a hostname and never reads
# cluster-vars — there is no second place the domain is decided.
#
# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
# -----------------------------------------------------
# The enrolment required action, `webauthn-register-passwordless`. It is
# deliberately NOT declared in the realm import, and the import's own header
# explains at length why declaring it there would be actively DANGEROUS: Keycloak
# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
# so declaring one action imports exactly that action plus delete_account and skips
# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
# lock-out of every platform-admin, because auth-otp-form in the browser flow still
# demands a TOTP they could then never enrol.
#
# So the required action cannot be copied from the import, because it is correctly
# absent from it. It is instead DERIVED, as a consequence of what the import DOES
# declare:
#
# IF the declared browser flow contains a `webauthn-authenticator-passwordless`
# execution, THEN passkey logins are offered, THEREFORE the enrolment action must
# be registered and enabled — and NOT a defaultAction, because the import's whole
# stated posture is "additive and optional" and a defaultAction forces enrolment
# on every user at next login.
#
# This is a derivation, not a second declaration: there is no value here that a
# human could edit into disagreement with the import. If the import stops declaring
# the passkey execution, this renderer stops emitting the required action (and
# fails, because then the whole desired state is empty of passkey intent).
#
# WHAT IS DELIBERATELY NOT EXPORTED
# * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
# `webauthn-authenticator` two-factor authenticator; the passwordless
# authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
# into the two-factor family is a silent no-op. Exporting it would hand the
# reconciler a write surface with no purpose.
# * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
# SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
# and users/groups/roles are nobody's — prod's real user and the media-users /
# platform-admins / viewers groups must survive every run of everything.
# * execution `priority`. It is consumed here to ORDER the exported executions
# (Keycloak returns them in priority order), but the numbers themselves are not
# exported: the reconciler compares the resulting SEQUENCE, which is what
# actually determines behaviour. Exporting the integers would make a
# priority renumbering that changes no ordering read as drift.
#
# USAGE
# python3 ci/render-keycloak-passkey-policy.py # print the rendered file
# python3 ci/render-keycloak-passkey-policy.py --write # write it (regenerate)
# python3 ci/render-keycloak-passkey-policy.py --check # exit 1 if it has drifted
# =============================================================================
from __future__ import annotations
import argparse
import difflib
import json
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(__file__).resolve().parent.parent
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
DATA_KEY = "passkey-desired-state.json"
CONFIGMAP_NAME = "keycloak-passkey-desired-state"
NAMESPACE = "keycloak"
# The realm keys this renderer exports, and therefore the ONLY realm keys the
# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
# tuples are identical, so they cannot drift apart silently.
POLICY_STR_FIELDS = (
"webAuthnPolicyPasswordlessRpId",
"webAuthnPolicyPasswordlessRpEntityName",
"webAuthnPolicyPasswordlessRequireResidentKey",
"webAuthnPolicyPasswordlessUserVerificationRequirement",
"webAuthnPolicyPasswordlessAuthenticatorAttachment",
"webAuthnPolicyPasswordlessAttestationConveyancePreference",
)
POLICY_LIST_FIELDS = (
"webAuthnPolicyPasswordlessSignatureAlgorithms",
"webAuthnPolicyPasswordlessAcceptableAaguids",
)
POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+ POLICY_BOOL_FIELDS)
# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
# (DefaultExportImportManager.updateRealm: the policy is applied only when
# RpEntityName is non-null and non-empty). A desired state without it would produce
# an accepted PUT that changes nothing, so refuse to render one.
RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
# The execution whose presence is what makes passkey enrolment necessary.
PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
ENROL_ACTION = "webauthn-register-passwordless"
BIND_KEY = "bind-browser-flow"
BIND_VALUES = ("true", "false")
def committed_bind(root: Path) -> str:
"""The operator's bind-browser-flow value from the COMMITTED file, or "false".
This is the only input read from the output file, and it is validated, not trusted:
anything other than the exact strings "true" / "false" is a hard failure."""
target = root / OUT_FILE
if not target.is_file():
return "false"
docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
data = cm.get("data") or {}
if BIND_KEY not in data:
return "false"
v = data[BIND_KEY]
if not isinstance(v, str) or v not in BIND_VALUES:
raise SystemExit(
"FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
"\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
"at apply time and which would fail the whole platform-config Kustomization"
% (OUT_FILE, BIND_KEY, v))
return v
def render(root: Path) -> str:
"""Return the full text of passkey-desired-state.yaml as it should be committed."""
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
if imp is None:
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
realm = (imp.get("spec") or {}).get("realm") or {}
realm_name = realm.get("realm")
if not isinstance(realm_name, str) or not realm_name:
raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
# ---- the passwordless policy, copied VERBATIM ----------------------------------
policy: dict[str, object] = {}
for f in POLICY_STR_FIELDS:
if f not in realm:
continue # not declared => NOT MANAGED
v = realm[f]
# A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
# a YAML quoting mistake that must not be silently stringified.
if not isinstance(v, str):
raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
"quote it in the realm import" % (IMPORT_FILE, f, v))
policy[f] = v
for f in POLICY_LIST_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
raise SystemExit("FATAL: %s realm field %s is not a list of strings"
% (IMPORT_FILE, f))
policy[f] = list(v) # verbatim, order as authored
for f in POLICY_INT_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, int) or isinstance(v, bool):
raise SystemExit("FATAL: %s realm field %s is not an integer"
% (IMPORT_FILE, f))
policy[f] = v
for f in POLICY_BOOL_FIELDS:
if f not in realm:
continue
v = realm[f]
if not isinstance(v, bool):
raise SystemExit("FATAL: %s realm field %s is not a boolean"
% (IMPORT_FILE, f))
policy[f] = v
if not policy:
# Fail closed. An empty policy would make the reconciler believe every field is
# unmanaged and silently stop reconciling — the realm import's own fail-open.
raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
"refusing to render an empty desired state" % IMPORT_FILE)
if not policy.get(RP_ENTITY_NAME):
raise SystemExit(
"FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
"passwordless policy ONLY when that field is set, so every realm PUT would "
"be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
# ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
browser_flow = realm.get("browserFlow")
if not isinstance(browser_flow, str) or not browser_flow:
raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
"be perfect and still not be the one anyone logs in through"
% IMPORT_FILE)
raw_flows = realm.get("authenticationFlows") or []
if not isinstance(raw_flows, list) or not raw_flows:
raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
flows: dict[str, object] = {}
for fl in raw_flows:
alias = fl.get("alias")
if not isinstance(alias, str) or not alias:
raise SystemExit("FATAL: %s has an authentication flow with no alias"
% IMPORT_FILE)
if alias in flows:
raise SystemExit("FATAL: %s declares the flow %r twice"
% (IMPORT_FILE, alias))
execs = fl.get("authenticationExecutions") or []
if not isinstance(execs, list) or not execs:
raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
% (IMPORT_FILE, alias))
ordered = []
for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
is_flow = bool(ex.get("authenticatorFlow"))
name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
if not isinstance(name, str) or not name:
raise SystemExit(
"FATAL: %s flow %r has an execution with neither an authenticator "
"nor a flowAlias" % (IMPORT_FILE, alias))
requirement = ex.get("requirement")
if not isinstance(requirement, str) or not requirement:
raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
% (IMPORT_FILE, alias, name))
entry: dict[str, object] = {
"kind": "flow" if is_flow else "authenticator",
"name": name,
"requirement": requirement,
}
cfg = ex.get("authenticatorConfig")
if cfg is not None:
if not isinstance(cfg, str) or not cfg:
raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
"authenticatorConfig" % (IMPORT_FILE, alias, name))
entry["configAlias"] = cfg
ordered.append(entry)
flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
if browser_flow not in flows:
raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
% (IMPORT_FILE, browser_flow))
# Every referenced subflow must be declared, or the reconciler's flattening walks
# into a hole and the audit silently covers less than it claims.
for alias, spec in flows.items():
for ex in spec["executions"]:
if ex["kind"] == "flow" and ex["name"] not in flows:
raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
"declared" % (IMPORT_FILE, alias, ex["name"]))
# ---- authenticator configs, copied verbatim ------------------------------------
configs: dict[str, dict] = {}
for cfg in realm.get("authenticatorConfig") or []:
alias = cfg.get("alias")
if not isinstance(alias, str) or not alias:
raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
% IMPORT_FILE)
body = cfg.get("config") or {}
if not isinstance(body, dict) or not body:
raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
% (IMPORT_FILE, alias))
# Values are stringified because Keycloak stores authenticator config values as
# strings. `negate: "false"` in the import and a stored "false" are the same
# declaration, and a YAML-unquoted `false` must not become a Python bool that
# then never matches.
configs[alias] = {k: str(v) for k, v in sorted(body.items())}
# Every configAlias a flow references must be defined, else the reconciler reports
# "git declares config X but defines no key/value for it" on every run forever.
for alias, spec in flows.items():
for ex in spec["executions"]:
ca = ex.get("configAlias")
if ca and ca not in configs:
raise SystemExit(
"FATAL: %s flow %r execution %r references authenticatorConfig %r, "
"which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
# ---- the enrolment required action, DERIVED (see the header) -------------------
offers_passkey = any(
ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
for spec in flows.values() for ex in spec["executions"])
if not offers_passkey:
raise SystemExit(
"FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
"in any flow. The policy decides what a passkey must BE; without that "
"execution nothing ever OFFERS one, so the policy is unreachable "
"configuration and this desired state would be meaningless"
% (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
want = {
"realm": realm_name,
"webAuthnPolicyPasswordless": policy,
# enabled: passkey enrolment must be possible, since the flow offers passkeys.
# defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
# enrol a passkey at next login — the platform-wide lock-out the realm import's
# "additive and optional" posture exists to avoid. Derived, not copied; see the
# header for why the import cannot declare this itself.
"requiredAction": {"alias": ENROL_ACTION, "enabled": True,
"defaultAction": False},
"browserFlow": browser_flow,
"flows": flows,
"authenticatorConfig": configs,
}
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+ " %s: |\n" % DATA_KEY + indented + "\n")
HEADER = """---
# =============================================================================
# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
#
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm)
# regenerate: python3 ci/render-keycloak-passkey-policy.py --write
# gated by: conformance/tests/l0-static/keycloak-passkey-reconciler
# (runs this renderer with --check on every merge)
#
# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
# the WebAuthn passwordless policy, the enrolment required action, and the declared
# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
# it and (a) converges the policy and the required action onto it, (b) builds the
# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
# writing the flow users are logging in through, and (c) BINDS that verified flow as
# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
#
# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
# operator decision, carried through by the renderer from this committed file (see
# ci/render-keycloak-passkey-policy.py). It defaults to "false".
#
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
# once Done=True, so a realm-level field or a flow edited in git never reaches the
# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
# Done=True / Started=False, with the whole passkey block inert. The reconciler is
# the level-triggered loop that closes that; this file is what it reconciles
# TOWARDS.
#
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
# user-verification requirement and the flow tree would be a second source of truth,
# free to disagree with the realm import — the same "two declarations that can
# disagree" shape as the bug being closed. It is rendered from the import and the
# render is gated, so the two cannot drift.
#
# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
# substitution token exactly as the realm import writes it. Both files are rendered
# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
# same value in both and the domain is decided in exactly one place.
#
# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
# nine more (a hard lock-out of every platform-admin). It is derived instead from
# the presence of a webauthn-authenticator-passwordless execution in the flow:
# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
# false, so enrolment stays opt-in.
#
# A field NOT declared on the realm in the import is simply ABSENT here, and the
# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: %s
namespace: %s
labels:
app.kubernetes.io/name: %s
app.kubernetes.io/part-of: keycloak
data:
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
BIND_BLOCK = """ # ---------------------------------------------------------------------------
# THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
# "false", QUOTED.
#
# "false" the reconciler builds and verifies the candidate browser flow and
# logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
# Whatever the realm is bound to -- including a rollback -- is left alone.
# "true" it binds the verified candidate in one atomic PUT, and keeps the
# realm bound to it: a binding changed by hand is changed back.
#
# GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
# "Passkeys"), then a PR setting this to "true".
# ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
# one write setting browserFlow back to the previous alias from the Job log.
# ---------------------------------------------------------------------------
%s: "%s"
"""
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
g = ap.add_mutually_exclusive_group()
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
g.add_argument("--check", action="store_true",
help="exit 1 if the committed file has drifted from the source")
args = ap.parse_args()
want = render(ROOT)
target = ROOT / OUT_FILE
if args.write:
target.write_text(want, encoding="utf-8")
print("wrote %s" % OUT_FILE)
return 0
if args.check:
if not target.is_file():
print("CHECK FAIL: %s does not exist. Run: python3 "
"ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
file=sys.stderr)
return 1
have = target.read_text(encoding="utf-8")
if have == want:
print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
return 0
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
"realm on a passkey policy the realm import does not declare, and build "
"the login flow from a tree nobody authored.\n"
"Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
fromfile="committed " + OUT_FILE,
tofile="rendered from " + IMPORT_FILE):
sys.stderr.write(line)
return 1
sys.stdout.write(want)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+21
View File
@@ -0,0 +1,21 @@
---
# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: buildkit
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: true
dependsOn:
- name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/buildkit
+61
View File
@@ -0,0 +1,61 @@
---
# Flux Kustomization: Cloudflare Tunnel connector — reconciles ./platform/cloudflared.
#
# ISOLATION IS THE POINT OF THIS FILE. The Deployment it applies CANNOT become Ready until the
# operator enrols the tunnel (runbooks/cloudflare-tunnel.md): it ships with REPLACE-ME credentials
# and a placeholder UUID, and a wrong or revoked credential later produces the same state. So:
#
# * ITS OWN Kustomization, not a line in platform/kustomization.yaml. `platform` is wait:true and
# gates platform-config, which gates every apps wave; an unenrolled tunnel there would be a
# platform-wide stall ON MERGE, with the cause (a missing manual step) invisible from any Flux
# condition. Same reasoning, and same failure shape, as vpn-ingress's unfilled server key.
# * wait: false. Its pods are legitimately not-Ready whenever the credentials are absent or the
# edge is unreachable. Health is reported by the CloudflaredTunnelDown / ReplicasDegraded
# alerts (platform/cloudflared/alerts.yaml), not by this Kustomization's Ready condition —
# which reaches Ready on APPLY success.
# * NOTHING dependsOn THIS, and nothing should. A missing or invalid credentials Secret can then
# only ever affect the tunnel itself. conformance/tests/l0-static/cloudflared-ingress-rules
# fails the gate if any Kustomization gains `dependsOn: cloudflared`, or if this one stops
# being wait:false.
# * No Certificate, no HelmRelease, no CRD: nothing in the path for Flux's health checks to wait
# on even if wait were flipped (the Certificate-stall lesson).
#
# dependsOn: [platform-config] — A REAL EDGE.
# 1. CRD ORDERING: PodMonitor and PrometheusRule are monitoring.coreos.com CRs whose CRDs are
# installed by kube-prometheus-stack in `platform`; platform-config dependsOn platform, so this
# edge carries that guarantee transitively (Flux aborts a whole pass on an unknown kind).
# 2. THE FAR SIDE OF THE TUNNEL: the Gateway (and so the cilium-gateway-shared Service the ingress
# rules point at) is applied by platform-config. A tunnel that comes up before its origin is a
# support call.
#
# ZERO EFFECT ON LIVE TRAFFIC UNTIL DNS MOVES. Cloudflare only sends a hostname through a tunnel
# when that hostname's DNS record is a CNAME to <uuid>.cfargotunnel.com. This change creates no
# such record and leaves external-dns, the Gateway annotations, Pi-hole and CoreDNS untouched.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cloudflared
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/cloudflared
# REQUIRED, not boilerplate: platform/cloudflared/config.yaml renders `${BASE_DOMAIN}` into every
# ingress hostname, and alerts.yaml into both runbook_url annotations. Without this block the
# token becomes the EMPTY STRING and the Kustomization still reports Ready (substitution fails
# OPEN), leaving ingress rules for `plex.` and `*.` — a tunnel that 404s everything at best.
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS
# Kustomization by name with a `decryption:` stanza for cloudflared-credentials.sops.yaml.
# Without it the Secret applies cleanly full of ENC[...] and cloudflared cannot authenticate.
+8
View File
@@ -78,6 +78,13 @@ resources:
# `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
# stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
- vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
# Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge) — POST-platform tier
# (dependsOn platform-config, wait:false), and NOTHING dependsOn it. Non-gating is load-bearing,
# as for vpn-ingress: the Deployment cannot become Ready until the operator enrols the tunnel
# (runbooks/cloudflare-tunnel.md), so in a gating tier an unfinished manual step — or a revoked
# credential later — would stall every apps wave. Carries cloudflared-credentials.sops.yaml, so
# both overlays add a SOPS decryption patch for it (by name). See clusters/base/cloudflared.yaml.
- cloudflared.yaml # Flux Kustomization -> ./platform/cloudflared (dependsOn platform-config)
# PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
# change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
# wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
@@ -98,4 +105,5 @@ resources:
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
- buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
- cluster-vars.yaml # overlay knob surface (base defaults)
+17
View File
@@ -140,6 +140,23 @@ patches:
provider: sops
secretRef:
name: sops-age
# cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
# platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
# secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
# SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
# authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
# (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cloudflared
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
# account and the Anthropic subscription token from `claude setup-token`. Without this
+17
View File
@@ -131,6 +131,23 @@ patches:
provider: sops
secretRef:
name: sops-age
# cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
# platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
# secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
# SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
# authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
# (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cloudflared
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
# account and the Anthropic subscription token from `claude setup-token`. Without this
+87 -1
View File
@@ -98,7 +98,10 @@ RUN set -eux; \
# --- runtime -----------------------------------------------------------------
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
ARG CHAINSAW_VERSION=
ARG KUBECTL_VERSION=
@@ -198,3 +201,86 @@ WORKDIR /workspace
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
# debugging the image locally.
CMD ["/bin/bash"]
# =============================================================================
# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
# because CI builds on rootless BuildKit where no container runtime exists.
#
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
# * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
# * no network ........... exact: RUN --network=none, and the first check proves it.
# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
# Approximated by permission: everything uid 65532 can write must lie
# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
# /results) are writable — the image points HOME and TMPDIR there — and
# readOnlyRootFilesystem is enforced by
# platform/kyverno/admission/conformance-trigger-job-shape.yaml.
# =============================================================================
FROM conformance-runner AS verify
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
echo "--- no network (the build-time stand-in for the Job's isolation) ---"
if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
fi
echo "confirmed: no network"
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
# greps stderr would pass a build where the guard had stopped rejecting.
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
echo "confirmed: / is not writable"
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
echo "confirmed: running as uid $(id -u), not root"
echo "CONFORMANCE-RUNNER VERIFIED"
VERIFY
+9 -4
View File
@@ -61,10 +61,15 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
> digest-pinned to the conformance-runner image, `command` exactly
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
prints the exact `podman push --digestfile …` command for the operator instead. See
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
verified image under `sha-<commit>` and `main`. Renovate then moves
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
digest in a pull request (that file is a sensitive path, so the bump needs an
administrator's `/approve-override`).
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
default verify stays offline so an air-gapped build is fully checked, not half-checked).
+51 -129
View File
@@ -1,43 +1,49 @@
#!/usr/bin/env bash
###############################################################################
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
#
# ./build-image.sh # build + verify + print the digest and push command
# ./build-image.sh # build the verify target, then the image
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
# # Opt-in because it needs the Gitea host reachable; the
# # default verify is offline so an air-gapped build is
# # still fully checked rather than half-checked.
# ENGINE=docker ./build-image.sh # build with docker instead of podman
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
#
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
# non-root uid the policy demands, and prints the exact `podman push` line for a human.
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
# this script (podman locally) and CI both build.
#
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
# pins the toolbox gate runs with. Nothing in this file invents a version.
#
# THE DIGEST CAVEAT — read it before pasting a digest into anything.
# The digest printed here is the LOCAL manifest digest. It is normally identical to the
# registry digest (podman pushes the manifest it already has rather than recompressing),
# but that is a property of the push, not a guarantee. So the push command below uses
# `--digestfile`: the digest the registry actually returns is the authoritative one, and
# that is the one that goes into conformance/versions.env.
# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
# conformance/versions.env.
###############################################################################
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
ENGINE="${ENGINE:-podman}"
TAG_OVERRIDE="${IMAGE_TAG:-}"
VERIFY=1
VERIFY_CLONE=0
PRINT_ARGS=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--print-build-args) PRINT_ARGS=1 ;;
--verify-clone) VERIFY_CLONE=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
@@ -48,8 +54,6 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
@@ -59,14 +63,21 @@ source "${TOOLCHAIN_ENV}"
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
# this Containerfile on rootless BuildKit.
BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- image name: fixed by the admission policy -------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# The tag is build convenience only (a human-readable handle for the push). Admission
# ignores it: only the digest is ever referenced from a Job.
IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
# admission ignores tags entirely: only the digest is ever referenced from a Job.
IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
log "Building ${IMAGE_REF}"
@@ -76,86 +87,31 @@ info " CHAINSAW ${CHAINSAW_VERSION}"
info " KUBECTL ${KUBECTL_VERSION}"
info " FEDORA ${FEDORA_VERSION}"
"${ENGINE}" build \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
"${HERE}"
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
LABEL_ARGS=()
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
ARG_FLAGS=()
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
# --- verify: the point of the whole exercise ---------------------------------
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
# filesystem — the shape the admission policy forces. A pass here is the only evidence
# this image is usable; a build that merely succeeded is not.
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
# container runtime — rootless BuildKit only). See the stage's header for what a build step can
# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
# that only greps stderr would pass a build where the guard had stopped rejecting.
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
# expected failure — without it the verify dies at the first negative case.)
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo "confirmed: / is read-only"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
'
log "Verifying the toolchain (Containerfile target: verify)"
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
log "Toolchain verified"
fi
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
# The negative cases above prove conformance-clone refuses bad input. This proves the
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
# rev-parse assertion is only an unexercised line of shell.
@@ -184,46 +140,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
log "conformance-clone verified against the real repository"
fi
# --- digest ------------------------------------------------------------------
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
# --- result ------------------------------------------------------------------
IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
log "Built ${IMAGE_REF}"
info "local manifest digest ${DIGEST}"
info "image id ${IMAGE_ID}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
cat <<EOF
================================================================================
NOT PUSHED. Publishing is an operator action — this script holds no registry
credential and will never acquire one.
OPERATOR: run these two commands, then paste the digest from the digestfile into
conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
${ENGINE} login gitea.home.keithjackson.dev
${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
${IMAGE_REF}
Then:
cat /tmp/conformance-runner.digest
# -> sha256:.... THIS is the authoritative digest, not the local one above.
# conformance/versions.env
CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
and so is any other repository path.
TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
\`${IMAGE_REPO}@sha256:*\` to be replaced with the
LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
That is a separate, human-reviewed PR against the security boundary.
================================================================================
EOF
info "image id ${IMAGE_ID}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
info "Renovate moves conformance/versions.env onto the registry-returned digest."
+105 -14
View File
@@ -14,7 +14,7 @@ Pure standard library (matches the harness's no-pip-dependency posture); the che
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
there; L0 checks to git + PyYAML).
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] [--jobs N] <test-dir> [<test-dir> ...]
Exit 0 iff every selected offline check passes.
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
@@ -43,13 +43,35 @@ Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing fai
scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
instead of once per hand-written stage.
PARALLELISM (--jobs, env CONF_OFFLINE_JOBS)
-------------------------------------------
The checks are independent processes, so they run in a bounded thread pool (one `bash` child
per worker). Default: min(8, usable CPUs); `--jobs 1` is the old serial runner exactly. Each
check's stdout+stderr is CAPTURED and only reported after every check has finished, in the
ORDER THE DIRS WERE GIVEN, so the log, the JUnit and the step summary are byte-for-byte
independent of which check finished first (only the trailing "[N.Ns]" wall-time on each
status line varies). The 0/1/3 aggregation runs over the same ordered case list as before.
Parallel-safety is a property of the CHECKS, not of this runner: every check must keep its
scratch state in a private mktemp/mkdtemp dir and must not write into the tree or a fixed
path. The audit that established that for the current set is recorded in
decisions/2026-11-08-parallel-l0-checks.md; a new check that needs shared state must say so.
Nested pools: the keycloak reconciler checks run their own mutation suites in a pool
(mutations.py, env MUTATION_JOBS). When this runner is itself parallel it exports a SMALLER
MUTATION_JOBS (unless the caller already set one), so the two suites share the CPUs the
other jobs-2 checks are not using instead of each taking min(8, CPUs). See inner_jobs().
"""
from __future__ import annotations
import argparse
import os
import re
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor
import xml.etree.ElementTree as ET
from pathlib import Path
from xml.dom import minidom
@@ -97,6 +119,46 @@ def collect_meta(test_yaml: Path, prefix: str):
return name, reqs
def usable_cpus() -> int:
"""CPUs this process may run on (the cgroup/affinity view, not the host's count)."""
try:
return max(1, len(os.sched_getaffinity(0)))
except (AttributeError, OSError):
return max(1, os.cpu_count() or 1)
def positive_int(text: str) -> int:
n = int(text)
if n < 1:
raise argparse.ArgumentTypeError(f"must be >= 1, got {n}")
return n
def inner_jobs(cpus: int, outer: int) -> int:
"""MUTATION_JOBS to hand the checks when this runner runs `outer` of them at once.
Two checks (the keycloak client-secret and passkey reconcilers) fan out internally; the
other outer-2 workers are single processes. Give the two suites an equal share of the CPUs
those others leave free, never fewer than 2 each (a 1-wide pool would make the 40-mutation
passkey suite the critical path of the whole stage), never more than 8. Peak process count
is then about (outer-2) + 2*inner, i.e. <= cpus on any machine with cpus >= outer + 2, and
at worst ~1.25-1.5x cpus on a small runner — CPU-bound oversubscription of that size costs
latency per process, not throughput.
"""
if outer <= 1:
return min(8, cpus)
return max(2, min(8, (cpus - outer + 2) // 2))
def run_check(check: Path, env: dict) -> tuple[int, str, float]:
t0 = time.monotonic()
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True,
stdin=subprocess.DEVNULL, env=env,
)
return proc.returncode, (proc.stdout + proc.stderr).strip(), time.monotonic() - t0
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--junit", type=Path, required=True)
@@ -108,26 +170,54 @@ def main() -> int:
"(makes this runner exit 3); repeatable")
ap.add_argument("--summary-out", type=Path, default=None,
help="write one Markdown line per case here, for the CI step summary")
ap.add_argument("--jobs", type=positive_int, default=None,
help="checks to run at once (default: env CONF_OFFLINE_JOBS, else "
"min(8, usable CPUs)); 1 = serial")
ap.add_argument("dirs", nargs="+", type=Path)
args = ap.parse_args()
strict_scopes = set(args.unwired_scope)
cases = [] # (name, reqs, dir, rc, output, scope)
cpus = usable_cpus()
if args.jobs is not None:
jobs = args.jobs
elif os.environ.get("CONF_OFFLINE_JOBS", "").strip():
try:
jobs = positive_int(os.environ["CONF_OFFLINE_JOBS"].strip())
except (ValueError, argparse.ArgumentTypeError) as e:
ap.error(f"CONF_OFFLINE_JOBS={os.environ['CONF_OFFLINE_JOBS']!r}: {e}")
else:
jobs = min(8, cpus)
child_env = dict(os.environ)
if not child_env.get("MUTATION_JOBS", "").strip():
child_env["MUTATION_JOBS"] = str(inner_jobs(cpus, jobs))
print(f"[offline] running {len(args.dirs)} check(s), jobs={jobs} "
f"(cpus={cpus}, MUTATION_JOBS={child_env['MUTATION_JOBS']})", file=sys.stderr)
# Metadata first, in the given order; then the checks themselves through the pool. The
# results are consumed in SUBMISSION order, so `cases` is identical to the serial runner's.
planned = [] # (name, reqs, dir, scope, check-or-None)
for d in args.dirs:
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
if check is None:
cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
scope))
continue
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True
)
out = (proc.stdout + proc.stderr).strip()
cases.append((name, reqs, d, proc.returncode, out, scope))
planned.append((name, reqs, d, scope, check))
cases = [] # (name, reqs, dir, rc, output, scope)
elapsed = [] # wall seconds per case, same order (None: nothing ran)
with ThreadPoolExecutor(max_workers=jobs) as pool:
futures = [pool.submit(run_check, check, child_env) if check is not None else None
for (_n, _r, _d, _s, check) in planned]
for (name, reqs, d, scope, check), fut in zip(planned, futures):
if fut is None:
cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
scope))
elapsed.append(None)
continue
rc, out, secs = fut.result()
cases.append((name, reqs, d, rc, out, scope))
elapsed.append(secs)
failures = sum(1 for c in cases if c[3] not in (0, 3))
skipped = sum(1 for c in cases if c[3] == 3)
@@ -145,7 +235,7 @@ def main() -> int:
},
)
summary_lines = []
for name, reqs, d, rc, out, scope in cases:
for (name, reqs, d, rc, out, scope), secs in zip(cases, elapsed):
tc = ET.SubElement(
suite,
"testcase",
@@ -164,8 +254,9 @@ def main() -> int:
fail.text = out
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
mark = f" scope={scope}" if scope else ""
took = f" [{secs:.1f}s]" if secs is not None else ""
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
f"({','.join(reqs) or 'untagged'}){mark}",
f"({','.join(reqs) or 'untagged'}){mark}{took}",
file=sys.stderr)
if rc not in (0, 3):
print(out, file=sys.stderr)
@@ -411,6 +411,32 @@ SCRAPE_OWNERS: list[dict] = [
"Keeping the two in one file makes that pair inseparable",
},
},
# ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ───────
# Purely additive, the same shape as the kured entry: it registers the cloudflared_* family
# and the Service+ServiceMonitor that scrapes it, and makes the check STRICTER for that component.
{
"owner": "cloudflared: headless metrics Service + ServiceMonitor on port `metrics` (:2000)",
"prefixes": ("cloudflared_",),
"file": "platform/cloudflared/servicemonitor.yaml",
"require": [
r"^kind:\s*ServiceMonitor\s*$",
r"^kind:\s*Service\s*$",
r"^\s*-\s*port:\s*metrics\s*$",
],
"forbid": [],
"why": "cloudflared_tunnel_ha_connections powers CloudflaredTunnelDown, the only signal "
"that the tunnel has lost the Cloudflare edge. That rule is written to FIRE on an "
"absent series, so a lost scrape is loud rather than inert — but it would then be "
"permanently firing for a wiring reason, which teaches the operator to ignore it",
"companion": {
"file": "platform/cloudflared/kustomization.yaml",
"require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
"why": "platform/cloudflared/networkpolicy.yaml ships the namespace's default-deny "
"(a platform namespace, outside Kyverno's generator) together with the ingress "
"allow for Prometheus on :2000. Keeping them in one file, applied together, is "
"what stops a baseline-without-allow from emptying the TSDB",
},
},
]
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
@@ -29,7 +29,8 @@
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
# pinned prometheus release and records it for provenance, build-image.sh (the one
# recipe `dev.sh build` and CI both use) passes it, and the
# CI runner installs it for the l0 stage;
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
# evidence only if the engine matches the one prod evaluates rules with;
@@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
"that provenance line is how an operator tells a stale image from a current one")
DEVSH = ROOT / "infra/devcontainer/dev.sh"
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
"empty and the build would fetch a bogus URL")
# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
"infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
"would be empty and the build would fetch a bogus URL")
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
if INSTALL.is_file():
@@ -0,0 +1,47 @@
# =============================================================================
# L0 STATIC — the Cloudflare Tunnel exposes only what the operator decided, and cannot stall Flux
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# platform/cloudflared/config.yaml is the entire internet-facing surface of the tunnel. This test
# asserts the operator decisions of 2026-11-08 against it on every gate run: no rule may target
# anything but a 404 or the shared Gateway (so never the Kubernetes API); the LAN/VPN-only
# hostnames (plex, jackett, pihole, semaphore, qbittorrent, radarr, sonarr, headlamp,
# unifi, crafty-mc) 404 before the wildcard; the
# egress policy admits no API/node entity; and the Flux tier is isolated (wait:false, nothing
# dependsOn it, both overlays decrypt it). It proves itself non-vacuous by mutation. Rationale and
# the honest NOT-proven list: check.sh.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans platform/cloudflared, platform/gateway (the
# derived Service name), clusters/base and both overlays. A requirement-scoped selection would skip
# it on, e.g., a Gateway rename — precisely a change that breaks it.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: cloudflared-ingress-rules
labels:
conformance.home.k3s/R-NET-01: "true"
conformance.home.k3s/R-NET-02: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-01, R-NET-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The Cloudflare Tunnel routes only to the shared Gateway or a 404, never to the Kubernetes
API; the LAN/VPN-only hostnames 404 ahead of the wildcard; the connector's egress admits no
API or node entity; and its Flux tier cannot stall any other tier.
spec:
steps:
- name: assert-tunnel-ingress-rules-and-isolation
description: >-
Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's
target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the
excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and
run the evaluator on mutated configs to prove it is not vacuous; and assert the
credentials Secret volume is readable by the container's run-as uid/groups (fsGroup +
mode), self-tested by mutation.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,321 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — cloudflared-ingress-rules
# Serves: R-NET-01 / R-NET-02 (default-deny; allow rules declarative and scoped), and the four
# operator decisions of 2026-11-08 recorded in decisions/2026-11-08-cloudflare-tunnel.md.
#
# WHY THIS EXISTS. platform/cloudflared/config.yaml is the whole of what the internet can reach
# through the Cloudflare Tunnel. A one-line edit there — a rule pointing at the Kubernetes API, a
# LAN-only hostname moved below the wildcard, a dropped catch-all — publishes something the operator
# decided must never be public, and nothing in the cluster turns red: the tunnel stays connected and
# the Kustomization stays Ready. So the decisions are asserted here, on every gate run.
#
# GIVEN the repository as authored,
# THEN (1) every ingress rule's `service` is `http_status:404` or EXACTLY the shared Gateway's
# in-cluster Service, derived from platform/gateway/gateway.yaml (so a Gateway rename
# fails here rather than in production) — which by construction excludes the
# Kubernetes API, and a named deny-list says so explicitly when it is violated;
# (2) each LAN/VPN-only hostname (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
# qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) has a rule mapping it to http_status:404 that comes BEFORE any
# Gateway-bound rule (cloudflared matches first-rule-wins);
# (3) the apex and the wildcard are the Gateway rules, with matchSNItoHost, and nothing
# disables origin TLS verification; the last rule is a hostname-less 404 catch-all;
# (4) the egress CiliumNetworkPolicy admits none of kube-apiserver/host/remote-node/
# cluster/all and no CIDR rule, and its backend selector excludes the LAN-only
# namespaces (the second, independent layer behind (2));
# (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in
# platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its
# SOPS decryption patch — so a missing or invalid credential can stall nothing else;
# (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
# (an API rule added, plex moved below the wildcard, the catch-all removed) and must
# flag each one;
# AND (7) every Secret volume the container mounts — the tunnel credentials above all — is
# READABLE by the container's run-as identity, modelled the way the kubelet lays it out:
# files owned by uid 0, group-owned by fsGroup if set (else gid 0), with the item mode or
# defaultMode (API default 0644). The identity is the container securityContext over
# the pod's; the process's groups are runAsGroup + fsGroup + supplementalGroups. An
# unset runAsUser is a FAIL, not a pass: the image default cannot be read offline. The
# model is deliberately conservative — it does not credit the kubelet's habit of
# OR-ing group-read into fsGroup-owned volumes, so 0400+fsGroup fails here though it
# would work. Self-tested by mutation (fsGroup removed; mode 0400; runAsUser unset).
# This is the 2026-11-08 CrashLoop: runAsUser 65532, no fsGroup, 0440 root:root.
#
# NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel,
# that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are
# runtime properties, verified by the operator per runbooks/cloudflare-tunnel.md.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import copy
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("CHECK FAIL (cloudflared-ingress-rules): PyYAML is required", file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
COMP = ROOT / "platform" / "cloudflared"
CONFIG = COMP / "config.yaml"
if not CONFIG.is_file():
print(f"CHECK FAIL (cloudflared-ingress-rules): {CONFIG.relative_to(ROOT)} is absent — the "
"component this check guards was moved or deleted; re-point or delete this check in the "
"SAME change", file=sys.stderr)
raise SystemExit(1)
# The hostnames the operator decided never go through the tunnel (decisions 2026-11-08).
# headlamp: operator decision 2026-11-08 (LAN/VPN-only). qbittorrent/radarr/sonarr: catalog
# `exposure: internal`, like jackett/pihole/semaphore.
EXCLUDED_HOSTS = ("plex", "plex-vip", "jackett", "pihole", "pihole-dns", "semaphore",
"qbittorrent", "radarr", "sonarr", "headlamp", "unifi", "crafty-mc")
# Their Gateway-backend namespaces (unifi / crafty-mc are L4 VIPs, not Gateway backends).
EXCLUDED_NAMESPACES = ("plex", "jackett", "pihole", "semaphore", "qbittorrent", "radarr",
"sonarr", "headlamp")
FORBIDDEN_ENTITIES = {"kube-apiserver", "host", "remote-node", "cluster", "all"}
# Any of these in a `service` means a route toward the Kubernetes API (or a raw-TCP/SSH bastion).
API_MARKERS = ("kubernetes", "kube-apiserver", "apiserver", ":6443", "10.43.0.1", "tcp://",
"ssh://", "unix:", "bastion")
# ── substitution: ${BASE_DOMAIN} from cluster-vars, the ONLY token allowed ──────────────────────
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
if d.get("kind") == "ConfigMap")
base_domain = cv["data"]["BASE_DOMAIN"]
raw = CONFIG.read_text()
tokens = set(re.findall(r"\$\{([A-Za-z0-9_]+)\}", raw))
check(tokens == {"BASE_DOMAIN"},
f"(0) config.yaml uses exactly the ${{BASE_DOMAIN}} token (found {sorted(tokens)}); any other "
f"token is substituted from cluster-vars unseen by this check")
check(re.search(r"\$(?!\{BASE_DOMAIN\})", raw) is None,
"(0) config.yaml contains a bare `$` — Flux envsubst would rewrite or empty it")
cfg = yaml.safe_load(raw.replace("${BASE_DOMAIN}", base_domain))
# ── the Gateway Service, derived rather than hard-coded ─────────────────────────────────────────
gw = next(d for d in docs(ROOT / "platform" / "gateway" / "gateway.yaml") if d.get("kind") == "Gateway")
GATEWAY_SVC = (f"https://cilium-gateway-{gw['metadata']['name']}."
f"{gw['metadata']['namespace']}.svc.cluster.local:443")
def evaluate(cfg):
"""Return the list of rule violations for a parsed cloudflared config."""
bad = []
rules = cfg.get("ingress") or []
if len(rules) < len(EXCLUDED_HOSTS) + 3:
bad.append(f"only {len(rules)} ingress rules — the scan is vacuous")
return bad
for i, r in enumerate(rules):
svc = str(r.get("service", ""))
if any(m in svc.lower() for m in API_MARKERS):
bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}, which "
f"is a route toward the Kubernetes API / a raw bastion — the API stays "
f"local-only (operator decision 2026-11-08)")
if svc not in ("http_status:404", GATEWAY_SVC):
bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}; only "
f"http_status:404 or {GATEWAY_SVC} are permitted")
if (r.get("originRequest") or {}).get("noTLSVerify"):
bad.append(f"rule {i} disables origin TLS verification (noTLSVerify)")
if (cfg.get("originRequest") or {}).get("noTLSVerify"):
bad.append("top-level originRequest disables origin TLS verification (noTLSVerify)")
gw_idx = [i for i, r in enumerate(rules) if r.get("service") == GATEWAY_SVC]
first_gw = min(gw_idx) if gw_idx else len(rules)
for h in EXCLUDED_HOSTS:
fqdn = f"{h}.{base_domain}"
hits = [i for i, r in enumerate(rules) if r.get("hostname") == fqdn]
if not hits:
bad.append(f"no rule for LAN/VPN-only host {fqdn}; it would fall through to the "
f"wildcard and reach the Gateway")
continue
i = hits[0]
if rules[i].get("service") != "http_status:404":
bad.append(f"{fqdn} maps to {rules[i].get('service')!r}, not http_status:404")
if i > first_gw:
bad.append(f"{fqdn}'s 404 rule (#{i}) comes AFTER the first Gateway rule (#{first_gw}); "
f"cloudflared is first-match, so the Gateway rule wins")
gw_hosts = {rules[i].get("hostname") for i in gw_idx}
if gw_hosts != {base_domain, f"*.{base_domain}"}:
bad.append(f"the Gateway-bound hostnames are {sorted(map(str, gw_hosts))}; expected "
f"exactly the apex and the wildcard")
for i in gw_idx:
if (rules[i].get("originRequest") or {}).get("matchSNItoHost") is not True:
bad.append(f"Gateway rule #{i} lacks originRequest.matchSNItoHost: true — the Gateway "
f"would not see the SNI that selects its wildcard certificate")
last = rules[-1]
if "hostname" in last or last.get("service") != "http_status:404":
bad.append("the last rule is not a hostname-less http_status:404 catch-all")
return bad
# (1)-(3) the real config
for v in evaluate(cfg):
check(False, f"(1-3) {v}")
check(str(cfg.get("credentials-file", "")).startswith("/etc/cloudflared/creds/"),
"(3) credentials-file points into the mounted Secret volume")
# (6) mutation self-test — the evaluator must flag each broken variant
mutants = {}
m = copy.deepcopy(cfg)
m["ingress"].insert(0, {"hostname": f"k8s.{base_domain}",
"service": "https://kubernetes.default.svc:443"})
mutants["API rule added"] = m
m = copy.deepcopy(cfg)
plex = next(r for r in m["ingress"] if r.get("hostname") == f"plex.{base_domain}")
m["ingress"].remove(plex)
m["ingress"].insert(len(m["ingress"]) - 1, plex)
mutants["plex moved below the wildcard"] = m
m = copy.deepcopy(cfg)
m["ingress"] = m["ingress"][:-1]
mutants["catch-all removed"] = m
for name, mut in mutants.items():
check(evaluate(mut), f"(6) self-test: the evaluator did NOT flag the mutant '{name}' — the "
f"check is vacuous")
# (4) the egress policy
np_docs = docs(COMP / "networkpolicy.yaml")
cnps = [d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy"]
check(len(cnps) == 1, f"(4) exactly one CiliumNetworkPolicy in networkpolicy.yaml (found {len(cnps)})")
for cnp in cnps:
for i, rule in enumerate(cnp["spec"].get("egress") or []):
ents = set(rule.get("toEntities") or [])
check(not (ents & FORBIDDEN_ENTITIES),
f"(4) egress rule {i} admits {sorted(ents & FORBIDDEN_ENTITIES)} — that reaches the "
f"Kubernetes API or the nodes; the API stays local-only")
check(not (rule.get("toCIDR") or rule.get("toCIDRSet")),
f"(4) egress rule {i} uses a CIDR — a CIDR can name the API VIP; use entities")
for ep in rule.get("toEndpoints") or []:
notin = [e for e in ep.get("matchExpressions") or []
if e.get("key") == "io.kubernetes.pod.namespace" and e.get("operator") == "NotIn"]
missing = set(EXCLUDED_NAMESPACES) - set(notin[0]["values"] if notin else [])
check(not missing,
f"(4) backend selector in egress rule {i} does not exclude {sorted(missing)}")
for d in np_docs:
if d.get("kind") == "NetworkPolicy" and d["metadata"]["name"] == "default-deny":
check(set(d["spec"].get("policyTypes", [])) == {"Ingress", "Egress"},
"(4) default-deny covers both directions")
break
else:
check(False, "(4) networkpolicy.yaml ships no default-deny baseline for the namespace")
# (5) Flux isolation
fk = next(d for d in docs(ROOT / "clusters" / "base" / "cloudflared.yaml")
if d.get("kind") == "Kustomization")
check(fk["spec"].get("wait") is False,
"(5) clusters/base/cloudflared.yaml must be wait: false — its pods are legitimately not-Ready "
"until the tunnel is enrolled")
check(fk["spec"].get("path") == "./platform/cloudflared", "(5) the Kustomization reconciles ./platform/cloudflared")
check("cloudflared.yaml" in (ROOT / "clusters" / "base" / "kustomization.yaml").read_text(),
"(5) clusters/base/kustomization.yaml lists cloudflared.yaml")
check("cloudflared" not in (ROOT / "platform" / "kustomization.yaml").read_text(),
"(5) platform/kustomization.yaml must not include cloudflared — that is the gating tier")
for f in sorted((ROOT / "clusters").rglob("*.yaml")):
for d in docs(f):
for dep in (d.get("spec") or {}).get("dependsOn") or []:
check(not (isinstance(dep, dict) and dep.get("name") == "cloudflared"),
f"(5) {f.relative_to(ROOT)} {d.get('metadata', {}).get('name')} dependsOn "
f"cloudflared — that hands an unenrolled tunnel the power to stall it")
for env in ("dev", "prod"):
text = (ROOT / "clusters" / env / "kustomization.yaml").read_text()
check(re.search(r"name:\s*cloudflared\s*\n\s*namespace:\s*flux-system\s*\n\s*spec:\s*\n"
r"\s*decryption:", text) is not None,
f"(5) clusters/{env}/kustomization.yaml carries the cloudflared SOPS decryption patch")
dep = next(d for d in docs(COMP / "deployment.yaml") if d.get("kind") == "Deployment")
check(dep["spec"].get("replicas", 1) >= 2, "(5) the Deployment runs at least 2 replicas")
check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.dev/egress-class")
== "internet",
"(4) the pods are egress-class=internet — vpn would put the edge connections behind PIA")
check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False,
"(4) the pods mount no ServiceAccount token")
# (7) the credentials Secret is readable by the run-as identity
def unreadable_secret_volumes(pod_spec):
"""Return a description per Secret volume the containers mount but cannot read."""
bad = []
psc = pod_spec.get("securityContext") or {}
vols = {v["name"]: v for v in pod_spec.get("volumes") or [] if "secret" in v}
for c in (pod_spec.get("initContainers") or []) + (pod_spec.get("containers") or []):
csc = c.get("securityContext") or {}
uid = csc.get("runAsUser", psc.get("runAsUser"))
gid = csc.get("runAsGroup", psc.get("runAsGroup"))
fs_group = psc.get("fsGroup")
groups = {g for g in [gid, fs_group, *(psc.get("supplementalGroups") or [])]
if g is not None}
for vm in c.get("volumeMounts") or []:
v = vols.get(vm["name"])
if v is None:
continue
sec = v["secret"]
if uid is None:
bad.append(f"container {c['name']} mounts Secret volume {vm['name']} but sets no "
f"runAsUser (pod or container) — readability cannot be proven offline")
continue
default = sec.get("defaultMode", 0o644)
modes = [it.get("mode", default) for it in sec.get("items") or []] or [default]
owner_gid = fs_group if fs_group is not None else 0
for mode in modes:
readable = ((uid == 0 and mode & 0o400) or (owner_gid in groups and mode & 0o040)
or mode & 0o004)
if not readable:
bad.append(f"container {c['name']} (uid {uid}, groups {sorted(groups)}) cannot "
f"read Secret volume {vm['name']} ({sec.get('secretName')}): files "
f"are 0:{owner_gid} mode {mode:04o} — set pod fsGroup to a group "
f"the container runs with, keeping 0440")
return bad
pod_spec = dep["spec"]["template"]["spec"]
creds_vol = [v for v in pod_spec.get("volumes") or []
if (v.get("secret") or {}).get("secretName") == "cloudflared-credentials"]
check(len(creds_vol) == 1, "(7) the Deployment mounts Secret cloudflared-credentials as a volume")
for v in unreadable_secret_volumes(pod_spec):
check(False, f"(7) {v}")
sec_mutants = {}
m = copy.deepcopy(pod_spec)
m["securityContext"].pop("fsGroup", None)
m["securityContext"].pop("supplementalGroups", None)
sec_mutants["fsGroup removed"] = m
m = copy.deepcopy(pod_spec)
for v in m["volumes"]:
if "secret" in v:
v["secret"]["defaultMode"] = 0o400
for it in v["secret"].get("items") or []:
it["mode"] = 0o400
sec_mutants["mode 0400"] = m
m = copy.deepcopy(pod_spec)
m["securityContext"].pop("runAsUser", None)
for c in m["containers"]:
(c.get("securityContext") or {}).pop("runAsUser", None)
sec_mutants["runAsUser unset"] = m
for name, mut in sec_mutants.items():
check(unreadable_secret_volumes(mut),
f"(7) self-test: the readability model did NOT flag the mutant '{name}' — vacuous")
if CHECKS < 25:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL (cloudflared-ingress-rules): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS (cloudflared-ingress-rules): {CHECKS} assertions")
PY
@@ -92,9 +92,10 @@
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
# each publishes so the claim is reviewable, which is all a static check can offer.
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
# rather than a routable address; that is an open operator decision recorded in
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
# - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
# Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
# `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
# then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
@@ -26,7 +26,20 @@
# KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
# (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
# ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
# (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
# prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
# (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
# platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
# (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
# DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
# (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
# names it, and the only `service`-source publishers of one are the declared L4 DNS-only
# names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
# tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
# (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
# - That the record is gone from Cloudflare. Live read; belongs beside
@@ -36,8 +49,11 @@
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
# This check owns the NAME axis only.
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
# - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
# proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
# returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
# honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
# Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
#
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
@@ -90,9 +106,18 @@ CHECKS = 0
# ControllerValue = "dns-controller" -> the value that CLAIMS an object
# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
#
# THE PREFIX IS NOT A CONSTANT OF NATURE. Every key external-dns reads is `<annotationPrefix><name>`,
# and the prefix is a runtime flag whose DEFAULT moved in v0.22.0 (#6424: alpha -> GA, "with no
# fallback"). On 2026-11-07 the chart bump to v0.23.0 made every key below invisible to the running
# controller while this check — which only asserted the keys were PRESENT — stayed green. (8) now
# ties ANNOTATION_PREFIX to the HelmRelease's explicit `annotationPrefix`.
ANNOTATION_PREFIX = "external-dns.alpha.kubernetes.io/"
CONTROLLER_KEY = ANNOTATION_PREFIX + "controller"
CONTROLLER_CLAIMED = "dns-controller"
HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
HOSTNAME_KEY = ANNOTATION_PREFIX + "hostname"
# Any key that looks like an external-dns annotation, under ANY prefix variant (alpha, GA, typos).
EXTERNAL_DNS_KEY_RE = re.compile(r"^external-dns(\.[a-z0-9-]+)*\.kubernetes\.io/")
def check(cond, msg):
@@ -129,9 +154,12 @@ def docs_of(p: Path):
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
#
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
# the honest state after this change, and it is deliberate: converting them is an operator
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
# WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
# PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
# address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
# remaining name should be public at all stays an operator decision per name, not a renderer change,
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
#
@@ -139,27 +167,28 @@ def docs_of(p: Path):
# =============================================================================================
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
"publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
"platform component, not a catalog service; it must be reachable by every browser that "
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
"whether the five public services are — an operator decision, not a renderer one.",
"platform/keycloak/httproute.yaml:keycloak/keycloak":
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
"publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
"is meant to be public either way; it is listed because the PUBLISHING object is the "
"hand-authored platform route, which carries no exposure field.",
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
"certainly should not be publicly resolvable, but it is not a catalog service and "
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
"internal services (Pi-hole must answer first).",
"publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
"behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
"name (it stayed public in the 2026-11-08 tunnel decision).",
"platform/headlamp/httproute.yaml:headlamp/headlamp":
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
"Keycloak OIDC; same adjudication as grafana.",
"publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
"decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
"with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
"disowned by (9d).",
"platform/homepage/httproute.yaml:homepage/homepage":
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
"publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
"declaration.",
"runbooks/serving/httproute.yaml:runbooks/runbooks":
@@ -168,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
"header for why gating runbooks on Keycloak is a loop with no exit.",
"platform/crafty/httproute.yaml:crafty/crafty":
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
"publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
"oauth2-proxy sidecar). Platform component, no exposure field.",
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
"Gateway cannot express. A game server is the one case where a public name may be "
"genuinely wanted; the address it answers with is still RFC1918.",
"genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
"(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
"vm-demo, which declares no ingressRoutes and publishes nothing.",
}
@@ -231,9 +260,7 @@ for sub in SCAN_DIRS:
ann = md.get("annotations") or {}
if kind == "HTTPRoute":
routes.append((rel, d))
elif kind == "Service" and any(
a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
):
elif kind == "Service" and any(EXTERNAL_DNS_KEY_RE.match(a) for a in ann):
dns_services.append((rel, d))
check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
@@ -444,6 +471,174 @@ check(
"and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
)
# =============================================================================================
# (8) THE RUNNING CONTROLLER READS THE PREFIX THIS TREE WRITES.
# (a) the external-dns HelmRelease sets `values.annotationPrefix` EXPLICITLY (an upstream
# default is exactly what moved under us) and it equals ANNOTATION_PREFIX;
# (b) it does not ALSO enable `--enable-legacy-annotation-prefix` (dual-read hides a
# half-migration; upstream says the flag will be removed);
# (c) every external-dns-looking annotation key on any object under apps/ platform/ runbooks/
# uses ANNOTATION_PREFIX — a key under another prefix is silently ignored at runtime;
# (d) catalog/render.py emits no external-dns key under another prefix.
# Migrating to the GA prefix = change ANNOTATION_PREFIX, the HelmRelease value, the renderer
# and every manifest in ONE commit; this assertion fails on any partial state.
# =============================================================================================
HR_PATH = ROOT / "platform/externaldns/helmrelease.yaml"
hr = next((d for d in docs_of(HR_PATH) if d.get("kind") == "HelmRelease"
and (d.get("metadata") or {}).get("name") == "external-dns"), None)
check(hr is not None, f"(8) {HR_PATH.relative_to(ROOT)} holds no HelmRelease `external-dns` — "
f"cannot prove which annotation prefix the controller reads")
values = ((hr or {}).get("spec") or {}).get("values") or {}
hr_prefix = values.get("annotationPrefix")
check(
hr_prefix == ANNOTATION_PREFIX,
f"(8a) external-dns HelmRelease `values.annotationPrefix` is {hr_prefix!r}, but every "
f"external-dns annotation in this tree uses {ANNOTATION_PREFIX!r}. A mismatch (or relying on "
f"the upstream default, which changed in v0.22.0) makes the controller IGNORE `controller: "
f"none`, the Gateway `target` and Service `hostname` — internal names go public, public names "
f"answer RFC1918 (incident 2026-11-07).",
)
extra = values.get("extraArgs") or []
extra_s = " ".join(f"{k}={v}" for k, v in extra.items()) if isinstance(extra, dict) \
else " ".join(map(str, extra))
check("enable-legacy-annotation-prefix" not in extra_s and "annotation-prefix" not in extra_s,
"(8b) external-dns extraArgs must not set --annotation-prefix or "
"--enable-legacy-annotation-prefix; the prefix is declared ONLY via values.annotationPrefix "
"so this check can see it")
foreign: list[str] = []
seen_keys = 0
for sub in SCAN_DIRS:
base = ROOT / sub
if not base.is_dir():
continue
for p in sorted(base.rglob("*.yaml")):
rel = p.relative_to(ROOT).as_posix()
for d in docs_of(p):
for k in annotations_of(d):
if isinstance(k, str) and EXTERNAL_DNS_KEY_RE.match(k):
seen_keys += 1
if not k.startswith(ANNOTATION_PREFIX):
foreign.append(f"{rel}:{'/'.join(ident(d))} {k}")
check(seen_keys > 0, "(8c) no external-dns annotation found anywhere — parse contract broken "
"(the Gateway target alone guarantees at least one)")
check(not foreign,
f"(8c) external-dns annotations under a prefix the controller does NOT read "
f"({ANNOTATION_PREFIX!r} is configured): {foreign}")
render_src = read(CATALOG / "render.py")
bad_render = sorted({m for m in re.findall(r"external-dns(?:\.[a-z0-9-]+)*\.kubernetes\.io/",
render_src) if m != ANNOTATION_PREFIX})
check(not bad_render,
f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
f"reads {ANNOTATION_PREFIX!r}")
# =============================================================================================
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
# =============================================================================================
CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
TARGET_KEY = ANNOTATION_PREFIX + "target"
PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
# deleting one from config.yaml cannot quietly also release it from (9c).
OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
cf = next(iter(docs_of(CF_CONFIG)), None)
check(cf is not None and isinstance(cf.get("ingress"), list),
f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
f"the tunnel refuses or which tunnel the Gateway must target")
cf = cf or {}
tunnel = str(cf.get("tunnel") or "")
check(bool(UUID_RE.match(tunnel)),
f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
f"the Gateway target is derived from it")
gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
and ident(d) == ("gateway", "shared")), None)
check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
want_target = f"{tunnel}.cfargotunnel.com"
got_target = annotations_of(gw or {}).get(TARGET_KEY)
check(got_target == want_target,
f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
f"an origin address of ours.")
check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
"(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
"CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
for rel, d in dns_services:
if is_disowned(d):
continue
if HOSTNAME_KEY not in annotations_of(d):
continue
v = annotations_of(d).get(PROXIED_KEY)
check(str(v).lower() == "false",
f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
for rel, d in routes:
v = annotations_of(d).get(PROXIED_KEY)
check(v is None or str(v).lower() == "true",
f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
"operator's LAN/VPN-only list is known to be non-empty)")
published_l4: set[str] = set()
for rel, d in routes:
hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
leaked = sorted(hosts & refused)
if is_disowned(d):
unrefused = sorted(hosts - refused)
check(not unrefused,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
continue
check(not leaked,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
for rel, d in dns_services:
if is_disowned(d):
continue
hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
if h.strip()}
for h in sorted(hosts & refused):
published_l4.add(h)
check(h in L4_DNS_ONLY_NAMES,
f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
check(L4_DNS_ONLY_NAMES <= refused,
f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
check(published_l4 == L4_DNS_ONLY_NAMES,
f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
f"published by any `service`-source publisher — delete them from the list")
check(OPERATOR_LAN_ONLY_NAMES <= refused,
f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
for name in sorted(OPERATOR_LAN_ONLY_NAMES):
naming = [(rel, d) for rel, d in routes
if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
for rel, d in naming:
check(is_disowned(d),
f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
# =============================================================================================
if FAILURES:
for f in FAILURES:
@@ -25,15 +25,19 @@
#
# python3 mutations.py <path-to-extracted-reconciler.py>
#
# Runtime is ~20s (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster).
# Runtime is ~11s serial (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
# verdicts are independent of the pool size.
# =============================================================================
from __future__ import annotations
import os
import re
import subprocess
import sys
import tempfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
@@ -136,43 +140,99 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
]
def mutation_jobs() -> int:
"""Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
running several checks in parallel, so the nested pools do not oversubscribe the host.
MUTATION_JOBS=1 reproduces the old serial loop exactly.
"""
raw = os.environ.get("MUTATION_JOBS", "").strip()
if raw:
try:
n = int(raw)
except ValueError:
n = 0
if n < 1:
raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
return n
try:
cpus = len(os.sched_getaffinity(0))
except (AttributeError, OSError):
cpus = os.cpu_count() or 1
return max(1, min(8, cpus))
def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(mutated_src)
path = fh.name
try:
return subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=240,
stdin=subprocess.DEVNULL)
finally:
Path(path).unlink(missing_ok=True)
def main() -> int:
if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2
src = Path(sys.argv[1]).read_text()
problems: list[str] = []
killed = 0
try:
jobs = mutation_jobs()
except ValueError as e:
print("mutations.py: %s" % e, file=sys.stderr)
return 2
# Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
# serial loop did, and each mutation's source is built here. None = anchor failure.
planned: list[tuple[str, str, str | None, str | None]] = []
for name, why, old, new in MUTATIONS:
old, new = dedent4(old), dedent4(new)
# An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything.
n = src.count(old)
if n != 1:
problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
planned.append((name, why,
"%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n))
"Re-anchor it against the current source." % (name, n), None))
continue
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(src.replace(old, new, 1))
path = fh.name
try:
proc = subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=240)
finally:
Path(path).unlink(missing_ok=True)
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-42s %s" % (name, how))
planned.append((name, why, None, src.replace(old, new, 1)))
# Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
# mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
# consumed in MUTATIONS order, so every line below is printed in the same order, with the
# same verdict, as the serial loop produced -- whichever child finishes first.
problems: list[str] = []
killed = 0
pool = ThreadPoolExecutor(max_workers=jobs)
try:
futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
for (_n, _w, _a, mutated) in planned]
for (name, why, anchor_problem, _m), fut in zip(planned, futures):
if anchor_problem is not None:
problems.append(anchor_problem)
continue
proc = fut.result()
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-42s %s" % (name, how))
finally:
# A timeout (or any other exception) still propagates and fails the check, as it did
# serially; queued mutations are cancelled rather than run for nothing.
pool.shutdown(wait=True, cancel_futures=True)
print()
if problems:
@@ -0,0 +1,108 @@
# =============================================================================
# L0 STATIC — the passkey configuration is derived from one declaration, continuously
# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
# Measured on prod 2026-10-18, hours after the passkey declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
# Done=True Started=False HasErrors=False
#
# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
# them changed nothing about how anyone logs in, and nothing anywhere said so.
# Declarative config that silently does not apply is not declarative config.
#
# GIVEN the repository as authored,
# THEN the reconciler's desired state is GENERATED from the realm import (so the RP ID —
# which is baked into every passkey at enrolment and can NEVER be re-keyed — is
# declared exactly once); both files are rendered by the SAME Flux Kustomization, so
# their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
# policy keys and the reconciler's writable keys are the SAME tuple; the realm
# import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
# delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
# enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
# no ServiceAccount token, mounts only what it needs, passes no mode flag, and
# contains no dollar sign; the endpoint allow-list contains NO delete, NO
# single-execution or config-edit write, and NO required-action write for any alias
# other than the one named literally; the protected-subtree guard exists, is wired
# into the single request choke point and is fail-closed; the SIBLING client-secret
# reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
# (extracted from the shipped ConfigMap and executed) proves compare-then-write,
# idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
# its sub-flows unwritable, the git bind switch and a rollback that sticks, the
# conditional-OTP regression and fail-closed — with mutations.py proving that
# selftest can FAIL.
#
# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
# than only argued: converging a five-flow, ten-execution restructure needs
# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
# calls with no transaction. Doing that to the BOUND flow means a run that dies between
# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
# which is verified by readback before anything is bound. Binding it is then ONE atomic
# PUT of browserFlow, made only when passkey-desired-state.yaml sets
# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
# a schedule tick, and with the switch off an operator's rollback is never undone.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
# Test wraps the SAME script.
#
# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
# password form (needs a real browser against a real realm); that the five flow-building
# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: keycloak-passkey-reconciler
labels:
conformance.home.k3s/R-IDP-03: "true"
conformance.home.k3s/R-IDP-04: "true"
annotations:
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
# check declares it: this invariant spans platform/ (the realm import, the reconciler,
# the generated desired state), platform-config/ (which Flux Kustomization renders
# them, and therefore which substitution pass they share), ci/ (the renderer) and
# clusters/ (that both are reachable from both entrypoints). A change that breaks it
# need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
# then filter this check out of exactly the change that introduces the bug — which is
# how 12 unusable prod OIDC clients sat behind an entirely green cluster on
# 2026-09-06.
conformance.home.k3s/scope: "whole-repo"
# Not safety-critical in the harness's sense (it mutates nothing and needs no
# cluster), but it guards a component that holds WRITE access to the identity
# provider's realm settings.
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The realm's passkey configuration — policy, enrolment action AND browser login
flow — is declared once in the realm import, derived from it by a gated renderer,
and continuously reconciled; the flow converges blue/green, so a one-shot import
that never re-runs can neither leave the configuration inert nor let a reconciler
half-rewrite the login path of the whole platform.
spec:
steps:
- name: audit-passkey-declaration-renderer-and-reconciler
description: >-
Assert the desired state is derived from the realm import and shares its
substitution pass, that the reconciler cannot write the bound flow or delete
anything and binds only on git's bind-browser-flow, that the sibling reconciler
is unwidened, and run the reconciler's own selftest plus the mutation suite that
proves it can fail.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,754 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
# — spec docs/15-identity-and-access.md)
# R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
# — spec docs/15-identity-and-access.md)
#
# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
# ---------------------------------------------------------------
# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
# hours after that declaration merged:
#
# kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
# Done=True Started=False HasErrors=False
#
# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
# re-runs it, so every realm-level field and every flow declared after the FIRST
# import is inert — in git, applied by Flux, reported Ready, reaching the realm
# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
# the same fix: a level-triggered reconciler. This check is the static half.
#
# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
#
# GIVEN the repository as authored,
# THEN (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
# (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
# import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
# renderer with --check, so a hand edit that makes the two disagree is
# unmergeable. The RP ID is the sharpest case: it is baked into every passkey
# at enrolment and can NEVER be re-keyed, so two declarations that can skew
# about the domain is an unrecoverable-credential bug, not a cosmetic one.
# (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
# postBuild.substituteFrom is per-Kustomization, so a split would let the
# ${BASE_DOMAIN} token in the realm import and the one in the desired state
# resolve to two different values — and an RP ID mismatch invalidates every
# enrolled passkey with no error the user can act on.
# (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
# declares no realm key outside the reconciler's write surface.
# (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
# reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
# could emit a key the reconciler refuses (fail-closed forever) or the
# reconciler could accept a key the renderer never emits (dead surface).
# (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
# entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
# ServiceAccount token, and contains no dollar sign (it ships inside a
# SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
# rewritten before any container runs).
# (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
# SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
# alias, verify, flip browserFlow in one atomic PUT), so flow-create and
# execution-add endpoints ARE on the allow-list and their presence is not the
# hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
# is the rollback target); the execution- and config-id-scoped write endpoints
# stay off the list because they carry no flow alias and so cannot be screened;
# no required-action write reaches any alias but the one named literally; and
# the guard the allow-list CANNOT express exists in the source and is wired
# into the single request choke point, FAIL-CLOSED until the live binding has
# been read. That last one is what the whole design rests on: a reconciler that
# half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
# this repo has had one (2026-09-06).
# (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
# bind-browser-flow as exactly "true" or "false", the script reads it from the
# SAME mount, and the CronJob passes no flag that could widen it. With "false"
# (the default) the reconciler builds and verifies but never PUTs browserFlow,
# so an operator's rollback sticks; the selftest proves that end to end.
# (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
# allow-list is NOT widened by this work. Its --selftest asserts it refuses
# /users, /groups, /roles, /authentication and the whole-realm endpoint, and
# its guarantee that prod's real user and the media-users / platform-admins /
# viewers groups survive every run must stay LITERALLY true.
# (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
# script from the ConfigMap and executes its `--selftest`, which drives the
# real code paths against an in-process stub Keycloak that MODELS the three
# Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
# passwordless policy, the all-or-nothing policy rebuild that defaults every
# omitted key, and the required-action representation replace.
# (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
# one at a time and requires --selftest to CATCH every one. A passing suite
# proves nothing if it cannot fail, and "reported success while the realm was
# broken" is the exact failure this component exists to end.
# AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
# `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
# via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
# ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
# An unreferenced test is precisely the "looks applied and isn't" failure this
# repo keeps hitting.
#
# NOT proven here (named honestly, in the style of the sibling L0 checks):
# - that the LIVE realm's policy, enrolment action or browser flow match git. That is
# cluster state. The reconciler's own `--audit` mode is the read-only probe for it
# and is what an L2 test should run; no such L2 test is added by this change, so the
# live realm is UNVERIFIED by CI.
# - that the declared browser flow actually WORKS as a login path. Nothing offline can
# prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
# password form, or that a platform-admin is still asked for TOTP. Those need a real
# browser against a real realm.
# - that the reconciler's first real run against prod will succeed. The flow is built
# through five admin-API endpoints whose request shapes were read from the 26.0.6
# source but never exercised against a live Keycloak.
# - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
# offline; the plaintext is not, and must not be.
# - that the CronJob's image can be pulled or its schedule fires.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
RENDERER="ci/render-keycloak-passkey-policy.py"
SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
# Honest SKIP (never green) if the thing under test is absent entirely.
for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
if [ ! -f "${ROOT}/${f}" ]; then
echo "check SKIPPED: ${f} absent — nothing to audit" >&2
exit 3
fi
done
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit of the Keycloak passkey declaration, the renderer that derives the
reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
is one definition of "what a Flux path renders" in the harness."""
from __future__ import annotations
import ast
import json
import re
import subprocess
import sys
import tempfile
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
# test directory is named relative to the repo root the wrapper passes in.
DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
/ "keycloak-passkey-reconciler")
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
import status as st # noqa: E402
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
RENDERER = "ci/render-keycloak-passkey-policy.py"
SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
STATE_CM_NAME = "keycloak-passkey-desired-state"
STATE_DATA_KEY = "passkey-desired-state.json"
BIND_KEY = "bind-browser-flow"
SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
ENROL_ACTION = "webauthn-register-passwordless"
DOLLAR = chr(36)
failures: list[str] = []
def check(cond, msg):
if not cond:
failures.append(msg)
def load(rel):
return list(yaml.safe_load_all((ROOT / rel).read_text()))
# ================================================== (1) the desired state is DERIVED
proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
capture_output=True, text=True, timeout=120)
check(proc.returncode == 0,
f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
f".py --check failed). The reconciler would converge the realm on a passkey policy "
f"the realm import does not declare, and audit the login flow against a tree "
f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
f"domain is an unrecoverable-credential bug.\n "
+ "\n ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
# The realm import must actually declare the things the renderer derives from. If it
# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
# renderer would otherwise be the only signal.
imp_docs = [d for d in load(IMPORT_FILE) if d]
imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
realm = ((imp or {}).get("spec") or {}).get("realm") or {}
policy_keys_in_import = sorted(k for k in realm
if k.startswith("webAuthnPolicyPasswordless"))
check(policy_keys_in_import,
f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
f"passkey policy for the reconciler to converge")
check(realm.get("browserFlow"),
f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
f"BOUND changes nothing about how anyone logs in")
# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
# representation as REPLACE, not merge, so declaring ONE action imports exactly that
# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
# NOT ADD ONE" is advice; this is the gate.
check("requiredActions" not in realm,
f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
f"as REPLACE, not merge: declaring any action imports exactly those plus "
f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
f"realm's browser flow still demands a TOTP they could no longer enrol. The "
f"enrolment action is already registered by default (priority 80, enabled, "
f"defaultAction false) and the passkey reconciler converges it through the "
f"per-alias admin API, which has no such hazard")
# ============================== (2)(3) the two files share one substitution pass
state_docs = [d for d in load(STATE_FILE) if d]
state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
desired = {}
if state_cm is not None:
check(state_cm["metadata"]["name"] == STATE_CM_NAME,
f"{STATE_FILE}: the ConfigMap is named "
f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
data = state_cm.get("data") or {}
check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
f"PASSKEY_BIND_FILE defaults would not resolve")
# THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
# at apply time -- failing the whole platform-config Kustomization, realm import
# included. The renderer refuses it too; asserted here so the reason is on record.
check(data.get(BIND_KEY) in ("true", "false"),
f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
f"string \"true\" or \"false\"")
raw = data.get(STATE_DATA_KEY, "")
try:
desired = json.loads(raw)
except ValueError as e:
check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
f"({e}) — the reconciler would fail closed on every run")
# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
import_owners = set()
state_owners = set()
recon_owners = set()
for env in ("dev", "prod"):
entry = f"clusters/{env}"
if not (ROOT / entry).is_dir():
continue
applied, _ = st.reconciled_files(ROOT, entry)
check(RECON_FILE in applied,
f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
f"healer would exist on one cluster only")
check(STATE_FILE in applied,
f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
f"would fail closed on that cluster forever")
check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
# listed as resources of the same kustomization.yaml.
for kpath in ROOT.glob("*/kustomization.yaml"):
try:
k = yaml.safe_load(kpath.read_text()) or {}
except Exception:
continue
res = [str(r) for r in (k.get("resources") or [])]
base = kpath.parent.name
if any(r.endswith("keycloak-realm-import.yaml") for r in res):
import_owners.add(base)
if any(r.endswith("passkey-desired-state.yaml") for r in res):
state_owners.add(base)
if any(r.endswith("passkey-reconciler.yaml") for r in res):
recon_owners.add(base)
check(import_owners and import_owners == state_owners,
f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
check(recon_owners == state_owners,
f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
f"{sorted(state_owners)} — the reconciler could be applied without its input")
# (3) no substitution token the realm definition does not also use.
import_text = (ROOT / IMPORT_FILE).read_text()
TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
import_tokens = set(TOKEN.findall(import_text))
# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
# policy uses, so a subset check is the right assertion rather than an exact match.
check(state_tokens <= import_tokens,
f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
f"silently — which for an RP ID means every passkey ceremony fails")
# ======================================= (4) the two write surfaces are the SAME tuple
recon_docs = [d for d in load(RECON_FILE) if d]
cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
script = ""
if cm is not None:
data = cm.get("data") or {}
check(len(data) == 1,
f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
f"(found {sorted(data)}) — a second copy would be untested by this gate")
check(SCRIPT_KEY in data,
f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
script = next(iter(data.values()), "")
check(len(script.splitlines()) > 200,
f"{RECON_FILE}: the extracted script is implausibly short "
f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
f"selftest below would prove nothing")
# THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
# postBuild.substituteFrom, so envsubst rewrites it before any container runs.
check(DOLLAR not in script,
f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
f"never needs one; chr(36) spells it where a literal is genuinely required")
def tuple_literal(src, name):
"""The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
than importing it keeps this check offline and side-effect free, and means the thing
asserted is the text that ships."""
try:
tree = ast.parse(src)
except SyntaxError:
return None
for node in tree.body:
if isinstance(node, ast.Assign):
for t in node.targets:
if isinstance(t, ast.Name) and t.id == name:
try:
return ast.literal_eval(node.value)
except ValueError:
return None
return None
renderer_src = (ROOT / RENDERER).read_text()
# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
# their parts from the same vocabulary; a part present in one and not the other simply
# contributes nothing, which is why the comparison below is of the resolved SETS.
def resolve_policy(src):
parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
"POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
"POLICY_BOOL_FIELDS")
out = []
for p in parts:
v = tuple_literal(src, p)
if v:
out.extend(v)
return sorted(set(out))
recon_keys = resolve_policy(script)
rend_keys = resolve_policy(renderer_src)
check(recon_keys and recon_keys == rend_keys,
f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
f"every run fails closed forever; if the reconciler accepts a key the renderer "
f"never emits, that is dead write surface on the identity provider")
# Every exported key must actually be a passwordless key. The two-factor
# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
# with no effect.
check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
f"the reconciler's writable keys include a non-passwordless field: "
f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
f"writing it is a silent no-op")
# And the desired state must declare only those.
if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
check(not extra,
f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
f"surface — it would fail closed on every run")
# The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
check(desired["webAuthnPolicyPasswordless"].get(
"webAuthnPolicyPasswordlessRpEntityName"),
f"{STATE_FILE} carries no non-empty "
f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
f"passwordless policy only when that field is set "
f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
f"accepted with 204 and change nothing")
# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
# import's stated "additive and optional" decision. It is one boolean, so it is gated.
ra = desired.get("requiredAction") or {}
check(ra.get("alias") == ENROL_ACTION,
f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
f"any other value is unreachable and the pass would fail closed")
check(ra.get("enabled") is True,
f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
f"impossible while the login flow offers passkeys")
check(ra.get("defaultAction") is False,
f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
f"must be false. True forces EVERY user to enrol a passkey at next login — this "
f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
# ========================================= (5) the reconciler manifest is safe + wired
if cj is not None:
pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
ctr = pod["containers"][0]
vols = {v["name"]: v for v in pod.get("volumes", [])}
mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
env = {e["name"]: e for e in ctr.get("env", [])}
# NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
# narrower than reading the KeycloakRealmImport CR through the API: no Role, no
# RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
check(pod.get("automountServiceAccountToken") is False,
f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
f"ServiceAccount token is pure blast radius (R-SEC-02)")
check("serviceAccountName" not in pod,
f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
check(STATE_CM_NAME in names,
f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
check(cm["metadata"]["name"] in names,
f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
# NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
# secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
check(not [v for v in vols.values() if "secret" in v],
f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
f"credential is referenced by secretKeyRef only; it must never be projected "
f"as a file here")
# The mount paths must be the ones the code DEFAULTS to, or the container starts and
# fails closed forever while every manifest looks correct.
state_vol = next((n for n, v in cm_vols.items()
if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
if state_vol:
mp = (mounts.get(state_vol) or {}).get("mountPath")
check(mp == "/etc/keycloak-passkey",
f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
f"script's PASSKEY_STATE_FILE default is "
f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
f"the mount path — they are a pair and must move together")
check((mounts.get(state_vol) or {}).get("readOnly") is True,
f"{RECON_FILE}: the desired-state mount must be readOnly")
check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
f"(safe, but git's \"true\" would then silently never take effect)")
check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
f"keys may not be projected")
# The bind decision belongs to git. A container arg could only NARROW it today
# (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
# behaviour must be readable from passkey-desired-state.yaml alone.
cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
check(not [x for x in cmd if x.startswith("--")],
f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
# Retries burn the bounded candidate-alias budget on a deterministic failure.
check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
f"within a tick, and each retried build consumes another candidate alias")
# The admin credential, referenced BY NAME. Never decrypted, never a literal.
for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
e = env.get(key) or {}
ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
check(ref.get("name") == "keycloak-admin-credentials",
f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
f"Secret by secretKeyRef, got {e!r}")
check("value" not in e,
f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
f"manifest is a plaintext secret in git")
check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
f"{RECON_FILE}: KEYCLOAK_REALM is "
f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
f"wrong realm, or none")
check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
f"{RECON_FILE}: readOnlyRootFilesystem must be true")
check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
f"{RECON_FILE}: allowPrivilegeEscalation must be false")
check(pod["securityContext"].get("runAsNonRoot") is True,
f"{RECON_FILE}: runAsNonRoot must be true")
check(cj["spec"].get("concurrencyPolicy") == "Forbid",
f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
f"would double the write attempts against the IdP during an outage recovery")
check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
#
# This component converges the browser flow BLUE/GREEN: it builds the target structure
# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
# first design of this check the mere presence of a flow write is no longer the thing to
# forbid. What must hold instead is narrower and stronger:
#
# (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
# (b) the single-execution and config-edit endpoints stay off the list — requirements
# are set through the flow-scoped endpoint, which the guard in (d) screens;
# (c) no required-action write may reach any alias but the one named LITERALLY;
# (d) and the guard the allow-list CANNOT express must exist in the source: a
# method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
# flow's subtree, FAIL-CLOSED until that binding has been read.
#
# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
# change (2026-09-06).
allow = tuple_literal(script, "_ALLOWED") or ()
check(bool(allow),
f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
f"script. This check's central assertions about the write surface cannot be made, "
f"so it must FAIL rather than pass silently")
if allow:
check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
f"allow-list cannot distinguish reading a flow from writing one, which this "
f"component's whole safety argument rests on")
writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
# (a) nothing is ever deleted.
check(not [(m, p) for m, p in allow if m == "DELETE"],
f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
f"does requires deleting anything: a superseded flow is left unbound as the "
f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
# (b) the endpoints that would let it edit live objects in place.
banned = [(m, p) for m, p in writes
if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
r"\[\^/\]\+", p)
or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
p)
or p.endswith("/authentication/register-required-action")]
check(not banned,
f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
f"and config-edit endpoints must stay off it: requirements are set through the "
f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
f"therefore screenable against the bound subtree. An execution- or "
f"config-id-scoped write is not, so it would be an unscreened hole")
# (c) no required-action write may reach any other alias.
ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
f"permanent lock-out of every platform-admin, because auth-otp-form in the "
f"browser flow still demands a TOTP that could no longer be enrolled")
check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
f"must be named LITERALLY so no other action is writable by construction")
# Nothing outside the realm + authentication surface at all.
stray = [(m, p) for m, p in allow
if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
check(not stray,
f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
f"be unreachable (prod's real user and the media-users / platform-admins / "
f"viewers groups must survive every run); clients are the SIBLING "
f"reconciler's surface and the two must not overlap")
# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
# Asserted against the SOURCE, because this is the property the blue/green design rests
# on and a comment claiming it is worth nothing. Three things must all be true.
check(re.search(r"^def _screen_write\(", script, re.M) is not None,
f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
f"one thing the allow-list cannot express, because its write entries are "
f"alias-parameterised. Without it this component can mutate the live login path")
# It must be CALLED from the single choke point, not merely defined.
check(re.search(r"_screen_write\(method, path", script) is not None,
f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
f"request's method and path. A guard that is not on the request path is decoration")
# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
# the top of every run must put them back there.
reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
check(reset_fn is not None
and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
f"must start each run UNDETERMINED, because while they are, every flow write is "
f"refused — that is the fail-closed half of the guard, and it is what stops a "
f"build running before the live binding has been read")
check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
f"protected sets are not reset per run and a previous run's binding could be "
f"trusted for this one")
# The create-flow endpoint carries NO alias in its path, so it must be screened on its
# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
# future edit cannot silently undo it.)
check("_FLOW_CREATE_PATH" in script,
f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
f"That endpoint carries no alias in its path — the alias is in the request BODY — "
f"so without a body-based screen it bypasses the protected-subtree guard "
f"altogether")
# And the one function allowed to change the live binding must be distinct from the
# policy write, which must refuse to.
# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
# the top-level flow alone while every selftest passed against a stub that returned it.
check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*? name = ex\.get\("displayName"\)',
script, re.M) is not None,
f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
f"None and the protected subtree silently shrinks to the top-level flow")
# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
f"and ignores the path alias, so a path-only screen lets it write the bound flow")
check("def declared_bind(" in script and "BIND_FILE" in script,
f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
check("def bind_browser_flow(" in script,
f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
f"one named, auditable operation, not an incidental field in some other write")
check(re.search(r"never rebinds the browser flow", script) is not None,
f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
f"can rebind the login path by accident — to an UNVERIFIED flow")
# ======================================== (7) the SIBLING reconciler is NOT widened
if (ROOT / SIBLING_FILE).is_file():
sib_docs = [d for d in load(SIBLING_FILE) if d]
sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
check(bool(sib_allow),
f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
f"prove the sibling was not widened")
if sib_allow:
# The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
# whole-realm endpoint and the /authentication tree; assert the same here so a
# future edit that "just adds the realm endpoint there instead" is caught.
bad = [p for p in sib_allow
if isinstance(p, str)
and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
or "authentication" in p or "/users" in p or "/groups" in p
or "/roles" in p)]
check(not bad,
f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
f"stated guarantee — prod's real user and the media-users / "
f"platform-admins / viewers groups survive every run — is expressed as "
f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
f"The passkey work is a SIBLING program with its own narrow list precisely "
f"so that guarantee stays literally true. Revert the widening")
# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
if script:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "reconcile.py"
p.write_text(script)
proc = subprocess.run([sys.executable, str(p), "--selftest"],
capture_output=True, text=True, timeout=300)
tail = (proc.stdout + proc.stderr).strip().splitlines()
check(proc.returncode == 0,
"the passkey reconciler's --selftest FAILED (compare-then-write / "
"idempotence / strict read-modify-write / the RpEntityName gate / "
"flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
" " + "\n ".join(tail[-14:]))
if proc.returncode == 0:
print(" passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
# ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
# A passing suite proves nothing unless it can be shown to FAIL. mutations.py
# injects plausible regressions one at a time -- a partial policy PUT, a
# rebind of browserFlow, a widened allow-list that makes the flow writable, a
# fail-open on an unreadable input -- and requires --selftest to catch every
# one. This matters more than usual here: the whole incident class is a
# component that REPORTED SUCCESS while the realm was broken, and a green test
# that cannot fail is the same lie one level up.
mut = DIR_PATH / "mutations.py"
if mut.is_file():
mproc = subprocess.run([sys.executable, str(mut), str(p)],
capture_output=True, text=True, timeout=900)
mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
check(mproc.returncode == 0,
"the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
"one injected regression survived it, so the suite does not prove "
"what it claims:\n " + "\n ".join(mtail[-16:]))
if mproc.returncode == 0:
print(" passkey reconciler mutations: "
+ (mtail[-1] if mtail else "(no output)"))
else:
check(False,
"conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
"is missing — nothing proves the --selftest can fail, and an "
"unfalsifiable proof is the failure mode this component exists to end")
# ======================================================= (9) this check is load-bearing
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
is not None,
"keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
"only when a change happens to attribute one of the requirements it is labelled "
"with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
"change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
"prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
"stage is a purely requirement-scoped run again and the scope annotation is inert "
"decoration — this check would then run only by coincidence")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
and re.search(r"fail\).*BLOCK=1", GATE),
"ci/gate.sh does not set the merge block when the l0 stage fails")
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
"ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
"import or the reconciler would attribute NO requirement and the keycloak tests "
"would be filtered out of the merge gate's touched subset")
# =============================================================================
if failures:
print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
for f in failures:
print(" FAIL %s" % f, file=sys.stderr)
raise SystemExit(1)
print("keycloak-passkey-reconciler: all assertions passed "
"(desired state derived from the realm import, one substitution pass, write "
"surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
"deletable, sibling allow-list unwidened, selftest green and falsifiable)")
PY
@@ -0,0 +1,516 @@
#!/usr/bin/env python3
# =============================================================================
# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
#
# WHY THIS EXISTS
# ---------------
# A selftest that passes proves nothing on its own: a selftest asserting `True ==
# True` also passes. This reconciler exists because a one-shot realm import
# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
# with the whole passkey block inert, measured on prod 2026-10-18), and the same
# trap applies one level up -- a green test suite that cannot fail is the same lie
# in a different place.
#
# It matters more here than for the sibling client reconciler, because this program
# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
# untested safety property on that surface is not a documentation gap, it is an
# outage waiting for a schedule tick.
#
# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
# a time, and requires that --selftest FAILS for every one. A mutation that
# SURVIVES means the suite does not actually check that property, and the check
# fails, naming it.
#
# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
# write -- never mutating the flow users are logging in through. Each of those
# properties is enforced in code; these mutations are the proof that the enforcement
# is load-bearing rather than commentary. Several model the obvious "simplification"
# someone will propose:
# * "just build it under the declared alias, the suffix is ugly"
# * "bind it first, then check -- the check is just belt and braces"
# * "the protected-alias screen is redundant, the allow-list covers it"
# Every one of those would be a total SSO outage waiting to happen.
#
# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
# The sibling mutations.py dedents by four because its anchors are copied out of the
# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
# which is one less transformation to get wrong. check.sh passes the extracted file,
# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
#
# python3 mutations.py <path-to-extracted-reconciler.py>
#
# Runtime is ~85s serial (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
# verdicts are independent of the pool size.
# =============================================================================
from __future__ import annotations
import os
import re
import subprocess
import sys
import tempfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
# (name, the mistake being modelled, anchor, replacement)
MUTATIONS: list[tuple[str, str, str, str]] = [
# ---------------------------------------------------------------- blue/green safety
("build INTO the currently bound flow",
"'the suffix is ugly, just build under the declared alias' -- every "
"create-execution and set-requirement call would then land on the flow users are "
"logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
"every SSO app and kubectl. This is the single regression the whole design exists "
"to make impossible",
" def mapped(alias):\n"
" return \"%s-%s\" % (alias, suffix)",
" def mapped(alias):\n return alias"),
("the protected-subtree screen is never called",
"'the allow-list already covers it' -- it does not: the allow-list's write entries "
"are alias-parameterised, so only this screen can say 'but not the flow that is "
"bound'. Without it the reconciler may write anywhere in the authentication tree",
" _screen_write(method, path, body if isinstance(body, dict) else None)",
" pass"),
("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
"reset them to frozenset() so the code reads more simply -- it silently removes "
"the fail-closed property, so a build that ran before the live binding had been "
"read would be PERMITTED, and could write straight into the live flow. (The "
"module-level initial value is deliberately NOT the anchor here: run() resets "
"these at the start of every run, so the initialiser is unreachable and mutating "
"it would prove nothing.)",
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
" _PROTECTED_FLOW_ALIASES = None\n"
" _PROTECTED_EXECUTION_IDS = None",
" global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
" _PROTECTED_FLOW_ALIASES = frozenset()\n"
" _PROTECTED_EXECUTION_IDS = frozenset()"),
("bind BEFORE verifying the build",
"'bind it first, the readback is just belt and braces' -- it is not: the readback "
"is what catches a build that was accepted but came out wrong, including the "
"conditional-OTP config landing under the wrong key. Binding first makes that the "
"live login path before anyone notices",
" verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
" \"the newly built flow %r\" % target)",
" verify = []"),
("a failed build is reported as SUCCESS",
"swallow the build error and return a converged result -- the realm is then left "
"on the old flow (which blue/green makes safe) but the run says it converged, so "
"nobody ever learns the passkey flow was never built. That is the one-shot "
"import's fail-open, reintroduced. NOTE: simply removing the early return here "
"is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
"is defence in depth working as intended -- so the mutation targets the reporting",
" problems.append(\n"
" \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
" \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
" \"place, unbound and harmless\" % (candidate, e))\n"
" return 0, problems",
" return 1, []"),
("an existing candidate flow is reused WITHOUT verifying it",
"'if the alias exists we already built it' -- a half-built leftover from a crashed "
"run lives under exactly that alias, so this binds a partially constructed login "
"path",
" if cand_shape is not None and not compare_shape(\n"
" cand_shape, expected, base, realm, token, configs,\n"
" \"the existing flow %r\" % candidate):",
" if cand_shape is not None:"),
# ------------------------------------- 2026-10-29 review findings, each made falsifiable
("sub-flows identified by `alias` again",
"read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
"recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
"sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
"a write into browser-mfa-forms would NOT be refused",
" name = ex.get(\"displayName\")\n"
" return name if isinstance(name, str) and name else None",
" name = ex.get(\"alias\")\n"
" return name if isinstance(name, str) and name else None"),
("an unnameable sub-flow is tolerated",
"carry on when a sub-flow row has no displayName -- the protected subtree is then "
"SMALLER than the live tree, which is the fail-open the guard exists to prevent",
" if name is None:\n"
" return None, None",
" if name is None:\n"
" name = \"\""),
("the requirement PUT is screened on its path only",
"trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
"body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
"the path can carry a BOUND execution's id in the body",
" targets.append((\"execution\", (body or {}).get(\"id\")))",
" pass"),
("execution-addressed writes are not held to the build tree",
"'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
"outside it too, so a stray id would edit an unrelated flow",
" if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
" or target not in _BUILD_EXECUTION_IDS):",
" if False:"),
("the requirement PUT may carry any key",
"let the body carry flowId / displayName -- the same handler then RENAMES whatever "
"flow flowId names, bound or not",
" if extra:\n"
" raise Refused(",
" if False:\n"
" raise Refused("),
("git's bind switch is ignored",
"bind whenever verification passes, as the first version did -- the first live use of "
"an untested login flow becomes a schedule tick, and an operator's rollback is "
"undone within 15 minutes, forever",
" bind = git_bind and not no_bind",
" bind = not no_bind"),
("bind=false still binds inside the flow pass",
"keep the flag but forget to honour it where the bind happens",
" if not bind:\n"
" # THE BIND SWITCH IS OFF.",
" if False:\n"
" # THE BIND SWITCH IS OFF."),
("an absent bind switch means bind",
"default to true when the key is missing -- a ConfigMap predating the switch, or a "
"deleted line, would silently put an untested flow live",
" except FileNotFoundError:\n"
" return False",
" except FileNotFoundError:\n"
" return True"),
("an invalid bind switch is guessed at",
"read anything that is not \"true\" as false -- a typo then passes silently instead of "
"stopping the run, and the operator believes the flow is live",
" if raw == \"false\":\n"
" return False\n"
" raise ValueError(",
" return False\n"
" raise ValueError("),
("bind=false rebuilds beside a hand-edited bound flow",
"treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
"off this builds an -r2 next to the operator's hotfix on every tick",
" if candidate in have and candidate == bound and not bind:",
" if False:"),
# ------------------------------------------------------------------- idempotence
("sub-flow aliases are compared as names",
"compare the sub-flow alias like any other field -- the flows this program builds "
"carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
"converged: it would be rebuilt and the login path reflipped every fifteen "
"minutes, forever",
" out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
" out.append((level, ex[\"kind\"], ex[\"name\"],\n"
" ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
("the converged short-circuit is removed",
"drop the 'bound flow already matches git' early return -- every run rebuilds and "
"reflips, which is a permanent write loop against the IdP and churns the realm's "
"login path on a schedule",
" if not drift:\n"
" log(\"flow ok (the bound flow %r matches git: %d execution(s), no \"\n"
" \"write)\" % (bound, len(expected)))\n"
" return 0, problems",
" if False:\n return 0, problems"),
("the execution priority is dropped from the requirement write",
"omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
"missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
"the execution order while only a requirement change was intended. The passkey "
"branch would end up in the wrong place in the tree",
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
" \"priority\": ex.get(\"priority\") or 0})",
" body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
("the alias budget is unbounded",
"loop forever looking for a free alias -- a persistently failing build then creates "
"unbounded flow sprawl in the realm, and nothing here can delete any of it",
" for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
" for attempt in range(1, 10000):"),
("fail-open when the bound flow cannot be read",
"carry on when the live flow's executions are unreadable -- the program then cannot "
"know which flows are live, so it cannot prove a build would miss them, and it "
"would build with the protected sets still undetermined",
" return 0, [\n"
" \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
" \"be read, so this program cannot establish which flows are live. Flow \"\n"
" \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
" % bound]",
" _set_protected(set(), set())\n bound_shape, bound_raw = None, None"),
# ------------------------------------------------------- the conditional-OTP regression
("authenticator config contents no longer compared",
"check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
"regression: the original bug wrote the role under "
"`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
"to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
"and every platform-admin got a 403 from every SSO-gated app",
" if str(have_cfg.get(key)) != str(want_cfg[key]):",
" if False:"),
("undeclared config keys are ignored",
"stop reporting keys the realm holds and git does not -- the stale "
"`condition.evaluated.user.role` key is the signature of the original null-role "
"bug, and a flow carrying both keys would look correct",
" extra = sorted(k for k in have_cfg if k not in want_cfg)",
" extra = []"),
("flow structure divergence detected but not reported",
"leave the per-step comparison out -- the realm's login flow could be the old "
"pre-passkey shape forever and every run would report the realm converged, which "
"is the exact fail-open of the one-shot import this program replaces",
" if observed[idx][:4] != expected[idx][:4]:",
" if False:"),
("the browserFlow BINDING is no longer compared",
"stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
"passkey flow that nothing logs in through changes nothing at all",
" if bound_shape is not None:\n"
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
" if False:\n"
" drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
# ------------------------------------------------------------- realm / policy writes
("partial policy PUT instead of read-modify-write",
"send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
"passwordless policy from the representation and substitutes its own DEFAULT for "
"every key the body omits, so a partial PUT silently resets everything it did not "
"mention while returning 204",
" body = dict(realm_rep)\n"
" for key in drift:\n"
" body[key] = wanted[key]",
" body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
" BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
" for key in drift:\n"
" body[key] = wanted[key]"),
("the policy pass rebinds browserFlow as a side effect",
"'while we are here, let the policy write fix the flow binding too' -- the realm "
"PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
"rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
"the passkey policy",
" if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
" raise RuntimeError(",
" body[BROWSER_FLOW] = \"browser\"\n"
" if False:\n"
" raise RuntimeError("),
("the RpEntityName gate removed from desired-state validation",
"accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
"passwordless policy ONLY when that field is non-empty, so every realm PUT would "
"be accepted with 204 and change nothing: reported success, zero convergence",
" if not policy.get(RP_ENTITY_NAME):\n"
" raise ValueError(",
" if False:\n raise ValueError("),
("the RpEntityName gate removed from the write path",
"drop the second, defence-in-depth guard -- it is unreachable while the "
"desired-state validation holds, which is exactly why deleting it would otherwise "
"be an invisible regression",
" if not body.get(RP_ENTITY_NAME):\n"
" raise RuntimeError(",
" if False:\n raise RuntimeError("),
("no readback verification on the policy write",
"trust the 204 -- a server that accepts the PUT and does not apply it is reported "
"as REPAIRED, which is 'reports success while the realm never converged' again, "
"and it is precisely what the RpEntityName gate does when it is missed",
" still = policy_drift(after, wanted)",
" still = {}"),
("AAGUID comparison made order-sensitive",
"compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
"is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
"realm on every run forever while every run still reports success",
" if stored != sorted(declared_value):\n"
" drift[key] = (stored, sorted(declared_value))",
" if stored != declared_value:\n"
" drift[key] = (stored, declared_value)"),
("signature-algorithm order treated as insignificant",
"sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
"to the authenticator, so ES256-then-RS256 is not the same declaration as "
"RS256-then-ES256, and a reorder in the realm would never be corrected",
" if stored != list(declared_value):",
" if sorted(stored) != sorted(declared_value):"),
("unmanaged policy keys forced to a value",
"iterate every writable key instead of only the declared ones -- a "
"webAuthnPolicyPasswordless field the realm holds and git does not declare would "
"be overwritten, turning 'not declared' into 'must equal my default'",
" for key in sorted(wanted):\n declared_value = wanted[key]",
" for key in POLICY_FIELDS:\n declared_value = wanted.get(key, \"\")"),
# ------------------------------------------------------------------ required actions
("required-action PUT sends only the two drifted fields",
"a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
"the representation, so name, priority and config are all blanked: the enrolment "
"action keeps working but loses its priority and display name",
" body = dict(rep)\n for key in drift:\n body[key] = wanted[key]",
" body = {}\n for key in drift:\n body[key] = wanted[key]"),
("the guarded default required actions are no longer audited",
"stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
"exist -- their disappearance is the signature of a requiredActions block in the "
"realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
"lock-out of every platform-admin",
" missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
" missing = []"),
# ----------------------------------------------------------------- inputs / coupling
("the three passes are coupled",
"abort the run at the first failing pass -- a policy write that fails for an "
"unrelated reason would then suppress the browser-flow pass, which is the most "
"safety-critical thing here",
" problems.extend(found)\n if found:\n rc = 1",
" problems.extend(found)\n if found:\n rc = 1\n"
" break"),
("policy-key validation removed from the desired state",
"accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
"writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
"bruteForceProtected or any other top-level realm field on the live IdP",
" if key not in POLICY_FIELDS:",
" if False:"),
("fail-open on an unreadable desired state",
"treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
"reconciling nothing, which is precisely the fail-open of the realm import this "
"whole component exists to replace",
" if not isinstance(raw, dict) or not raw:\n"
" raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
" if not isinstance(raw, dict) or not raw:\n return {}"),
("an undeclared authenticatorConfig reference is tolerated",
"stop validating that every configAlias a flow references is actually declared -- "
"the build then fails at the config-binding step, AFTER flows and executions have "
"already been created, leaving leftovers on every run",
" if ca and ca not in configs:\n"
" raise ValueError(",
" if False:\n raise ValueError("),
("the flow-cycle guard is removed",
"drop the `seen` check -- a cyclic flow declaration then recurses until the "
"interpreter dies, in a CronJob that retries",
" if alias in seen:\n"
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
" if False:\n"
" raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
]
def mutation_jobs() -> int:
"""Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
running several checks in parallel, so the nested pools do not oversubscribe the host.
MUTATION_JOBS=1 reproduces the old serial loop exactly.
"""
raw = os.environ.get("MUTATION_JOBS", "").strip()
if raw:
try:
n = int(raw)
except ValueError:
n = 0
if n < 1:
raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
return n
try:
cpus = len(os.sched_getaffinity(0))
except (AttributeError, OSError):
cpus = os.cpu_count() or 1
return max(1, min(8, cpus))
def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(mutated_src)
path = fh.name
try:
return subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=300,
stdin=subprocess.DEVNULL)
finally:
Path(path).unlink(missing_ok=True)
def main() -> int:
if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2
src = Path(sys.argv[1]).read_text()
try:
jobs = mutation_jobs()
except ValueError as e:
print("mutations.py: %s" % e, file=sys.stderr)
return 2
# Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
# serial loop did, and each mutation's source is built here. None = anchor failure.
planned: list[tuple[str, str, str | None, str | None]] = []
for name, why, old, new in MUTATIONS:
# An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything.
n = src.count(old)
if n != 1:
planned.append((name, why,
"%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n), None))
continue
planned.append((name, why, None, src.replace(old, new, 1)))
# Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
# mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
# consumed in MUTATIONS order, so every line below is printed in the same order, with the
# same verdict, as the serial loop produced -- whichever child finishes first.
problems: list[str] = []
killed = 0
pool = ThreadPoolExecutor(max_workers=jobs)
try:
futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
for (_n, _w, _a, mutated) in planned]
for (name, why, anchor_problem, _m), fut in zip(planned, futures):
if anchor_problem is not None:
problems.append(anchor_problem)
continue
proc = fut.result()
if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The "
"mistake it models: %s" % (name, why))
continue
killed += 1
out = proc.stdout + proc.stderr
m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)")
print(" killed %-54s %s" % (name, how))
finally:
# A timeout (or any other exception) still propagates and fails the check, as it did
# serially; queued mutations are cancelled rather than run for nothing.
pool.shutdown(wait=True, cancel_futures=True)
print()
if problems:
print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
% (len(problems), len(MUTATIONS)), file=sys.stderr)
for p in problems:
print(" FAIL %s" % p, file=sys.stderr)
return 1
print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
"(the suite is falsifiable, not decorative)" % killed)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,39 @@
# =============================================================================
# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it
# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0
# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header
# states the assertions and what they do not prove.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML);
# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-dns-source-ranges
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change
# to these Services would otherwise select nothing.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in
loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve
and PiHole stays unreachable from the internet.
spec:
steps:
- name: assert-pihole-source-ranges
description: >-
Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars,
and assert both required ranges are present and every range is private.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,146 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-dns-source-ranges
# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09).
#
# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being
# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions:
# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole
# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits
# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic).
# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel
# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables
# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the
# tunnel would still report a healthy handshake.
#
# GIVEN the repository as git tracks it,
# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range
# is resolved through cluster-vars (base value AND the prod overlay's replacement),
# THEN (1) the frontends exist and cover both udp/53 and tcp/53;
# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR};
# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public
# range, and no unresolvable token that could hide one;
# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count.
#
# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a
# handshake from a real client; see platform/vpn-ingress/README.md.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress, re, subprocess, sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR")
RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def docs_of(rel: str):
try:
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
if isinstance(d, dict)]
except Exception:
return []
FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True).stdout.split("\0")
if p]
YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)]
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
# ------------------------------------------------- every value a ${TOKEN} can take on this tree
values: dict[str, set[str]] = {}
for d in docs_of("clusters/base/cluster-vars.yaml"):
if d.get("kind") == "ConfigMap":
for k, v in (d.get("data") or {}).items():
values.setdefault(k, set()).add(str(v))
for d in docs_of("clusters/prod/kustomization.yaml"):
for p in d.get("patches") or []:
ops = p.get("patch") if isinstance(p, dict) else None
ops = yaml.safe_load(ops) if isinstance(ops, str) else ops
for op in ops if isinstance(ops, list) else []:
m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", "")))
if m and op.get("op") in ("add", "replace"):
values.setdefault(m.group(1), set()).add(str(op.get("value")))
for tok in REQUIRED_TOKENS:
check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)")
def assert_private(where: str, cidr: str):
try:
net = ipaddress.ip_network(cidr, strict=False)
except ValueError:
check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)")
return
check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918),
f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN "
"(R-DNS-03)")
# ---------------------------------------------------------------------------- the frontends
frontends = []
for rel in YAML_FILES:
for d in docs_of(rel):
md, spec = d.get("metadata") or {}, d.get("spec") or {}
if d.get("kind") == "Service" and md.get("namespace") == "pihole" \
and spec.get("type") == "LoadBalancer":
frontends.append((rel, md.get("name"), spec))
protocols = {(p.get("protocol", "TCP"), p.get("port"))
for _, _, spec in frontends for p in spec.get("ports") or []}
for want in (("UDP", 53), ("TCP", 53)):
check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this "
"check's corpus has moved and must be re-derived (R-DNS-03)")
for rel, name, spec in frontends:
where = f"{rel}: Service/{name}"
ranges = spec.get("loadBalancerSourceRanges") or []
check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach "
"the VIP (R-DNS-03)")
tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m}
for tok in REQUIRED_TOKENS:
check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges "
"(R-DNS-03)")
for r in ranges:
m = TOKEN.match(str(r))
if not m:
assert_private(where, str(r))
continue
resolved = values.get(m.group(1))
check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be "
"shown to be private (R-DNS-03)")
for v in sorted(resolved or []):
assert_private(f"{where} ({r})", v)
check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)")
if FAILURES:
for f in FAILURES:
print(f"check FAIL: {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over "
f"{len(frontends)} PiHole LoadBalancer Service(s)")
PY
@@ -0,0 +1,45 @@
# =============================================================================
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
# mutation. Rationale and the honest NOT-proven list: check.sh.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-lan-names
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
on the LAN path, off Cloudflare's request-size and timeout limits.
spec:
steps:
- name: assert-pihole-explicit-lan-names
description: >-
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
the evaluator on mutated lists to prove it is not vacuous.
try:
- script:
content: bash ./check.sh
+269
View File
@@ -0,0 +1,269 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-lan-names
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
#
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
#
# GIVEN the repository as authored,
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
# (b) every HTTPRoute hostname disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
# `hostnames:` is itself a FAIL, because it cannot be checked;
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
# the L4 DNS-only names (derived: Services carrying both
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
# limits break git push/LFS and large uploads);
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
# unifi repointed to the Gateway) and must flag each one.
#
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import copy
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
NAME = "pihole-lan-names"
ZONE = "${BASE_DOMAIN}"
GW = "${GATEWAY_VIP}"
DISOWN = "external-dns.alpha.kubernetes.io/controller"
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
LB_IPS = "lbipam.cilium.io/ips"
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
LAN_PINNED = ("gitea", "nextcloud")
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
# surface (CLAUDE.md §0); decisions/ is prose.
SCAN = ("apps", "platform", "runbooks")
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
def dnsmasq_value(objs):
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
found = []
for d in objs:
envs = list(d.get("env") or [])
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
for c in tmpl.get("containers") or []:
envs += c.get("env") or []
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
return found
def die(msg):
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
raise SystemExit(1)
CATALOG = ROOT / "catalog" / "pihole.yaml"
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
for p in (CATALOG, RENDERED, CF):
if not p.is_file():
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
"delete this check in the SAME change")
cat_vals = dnsmasq_value(docs(CATALOG))
ren_vals = dnsmasq_value(docs(RENDERED))
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(cat_vals)})")
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(ren_vals)})")
if not cat_vals:
die("no dnsmasq value to check")
check(cat_vals[:1] == ren_vals[:1],
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
if d.get("kind") == "ConfigMap")
CLUSTER_VARS = set(cv["data"])
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
def parse(value):
"""Return (entries {domain: target}, problems)."""
entries, problems = {}, []
for raw in value.split(";"):
line = raw.strip()
if raw != line:
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
if not line:
problems.append("empty entry (doubled or trailing `;`)")
continue
if re.fullmatch(r"local-ttl=\d+", line):
continue
m = ENTRY.match(line)
if not m:
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
f"this check cannot reason about it")
continue
dom, tgt = m.groups()
if tgt[2:-1] not in CLUSTER_VARS:
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
if dom in entries and entries[dom] != tgt:
problems.append(f"{dom} is listed twice with different targets")
entries[dom] = tgt
return entries, problems
entries, problems = parse(cat_vals[0])
for pr in problems:
check(False, f"(0) {pr}")
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
disowned, unnamed = set(), []
l4 = {}
for top in SCAN:
for f in sorted((ROOT / top).rglob("*.yaml")):
for d in docs(f):
meta = d.get("metadata") or {}
ann = meta.get("annotations") or {}
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
hosts = (d.get("spec") or {}).get("hostnames") or []
if not hosts:
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
disowned.update(hosts)
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
for h in str(ann[EDNS_HOST]).split(","):
l4[h.strip()] = str(ann[LB_IPS]).strip()
for u in unnamed:
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
f"hostname this check cannot see; name it explicitly")
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
f"is vacuous (the disown annotation was renamed?)")
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
f"unifi and crafty-mc) — the scan is vacuous")
cf = yaml.safe_load(CF.read_text())
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
if r.get("hostname") and r.get("service") == "http_status:404"}
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
f"config — the scan is vacuous")
def covers_zone(dom):
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
if dom in ("#", "", "."):
return True
if dom == ZONE:
return True
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
def evaluate(entries):
bad = []
# (a)
for dom in entries:
if covers_zone(dom):
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
f"on purpose (public names must resolve to Cloudflare from the LAN)")
# (b)
for h in sorted(disowned):
if entries.get(h) != GW:
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
f"unreachable everywhere")
# (c)
for h in sorted(cf_404):
if h in l4:
if entries.get(h) != l4[h]:
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
elif h not in entries:
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
f"only publicly, and the tunnel refuses it")
# (d)
for n in LAN_PINNED:
h = f"{n}.{ZONE}"
if entries.get(h) != GW:
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
f"100 s")
return bad
for v in evaluate(entries):
check(False, v)
# (e) mutation self-test
mutants = {}
m = dict(entries); m[ZONE] = GW
mutants["zone wildcard added"] = m
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
mutants["parent-of-zone wildcard added"] = m
m = dict(entries); m.pop(sorted(disowned)[0], None)
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
if pure_404:
m = dict(entries); m.pop(pure_404[0], None)
mutants[f"404'd host {pure_404[0]} dropped"] = m
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
mutants["gitea unpinned"] = m
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
mutants["nextcloud repointed"] = m
if l4:
h = sorted(l4)[0]
m = dict(entries); m[h] = GW
mutants[f"{h} repointed to the Gateway"] = m
for name, mut in mutants.items():
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
if CHECKS < 15:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
PY
@@ -42,6 +42,19 @@
# whose every solver is an automated challenge type (dns01 / http01) — the
# same code path serves issuance AND renewal, so a solver a human must
# complete would defeat both;
# (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
# annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
# requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
# ADR-0006 says outright that "mTLS between workloads is a separate concern … not
# covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
# client-auth certificate at all. The annotation is NOT trusted: an exempt
# Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
# listener, every DNS name / commonName is cluster-internal (a bare label or
# `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
# NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
# by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
# renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
# still applies. Every exemption is printed. First user: platform/buildkit.
# (6) each dns01 solver's provider credential is referenced BY NAME and that
# Secret is delivered SOPS-encrypted from git via a kustomization — the
# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
@@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName)
for rel, d in CERTS
}
listeners_checked = 0
GATEWAY_SECRETS: set[tuple[str, str]] = set() # every (namespace, Secret) a listener serves
for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
gw_ns = ns_of(gw)
for lis in ((gw.get("spec") or {}).get("listeners") or []):
@@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
for r in refs:
kind = str(r.get("kind", "Secret"))
key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
GATEWAY_SECRETS.add(key)
check(kind == "Secret",
f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
"Secret refs can be traced to a cert-manager Certificate")
@@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field)
issuers_verified: set[str] = set()
# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
for _, d in CERTS
if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
exempted: list[str] = []
def workload_mtls_ok(rel: str, cert: dict) -> bool:
"""True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
if ann is None:
return False
cns, cname = ns_of(cert), name_of(cert)
spec = cert.get("spec") or {}
ok = True
def need(cond, msg):
nonlocal ok
check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
ok = ok and bool(cond)
need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
"its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
names = [str(n) for n in (spec.get("dnsNames") or [])]
if spec.get("commonName"):
names.append(str(spec["commonName"]))
for n in names:
internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
for f in ("ipAddresses", "uris", "emailAddresses"):
need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
ref = spec.get("issuerRef") or {}
need(str(ref.get("kind", "Issuer")) == "Issuer",
"its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
if hit is not None:
ispec = hit[1].get("spec") or {}
kinds = sorted(ispec)
need(kinds in (["selfSigned"], ["ca"]),
f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
if kinds == ["ca"]:
need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
"its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
"in the same namespace — the CA itself would have to be minted by hand")
if ok:
exempted.append(f"{cns}/{cname}")
return ok
for rel, cert in CERTS:
spec = cert.get("spec") or {}
if workload_mtls_ok(rel, cert):
continue # (5b): proven workload-internal; (9) below still applies
ref = spec.get("issuerRef") or {}
kind = str(ref.get("kind", "Issuer"))
per_env = resolve_per_env(str(ref.get("name", "")))
@@ -379,6 +447,8 @@ for rel, cert in CERTS:
solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
if exempted:
print(f" (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
check(len(solver_creds) > 0,
@@ -62,7 +62,7 @@ spec:
ports:
- containerPort: 8080
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
@@ -85,7 +85,7 @@ spec:
- name: config
mountPath: /data/config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
@@ -85,7 +85,7 @@ spec:
- name: config
mountPath: /data/config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
@@ -64,6 +64,12 @@ spec:
annotations:
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
# DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
# record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
# and the exact-IP poll below can never match. Same opt-out every real
# `service`-source publisher carries (unifi, crafty-mc).
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec:
type: LoadBalancer
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
+18 -21
View File
@@ -40,32 +40,29 @@ PYTHON_MIN="3.9"
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# Build handle only. Admission never sees the tag; a Job must reference the digest.
# Local build handle only. Admission never sees the tag; a Job must reference the digest.
# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
#
# ############################################################################
# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. #
# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND. #
# ############################################################################
# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
# pushing needs a registry credential, and publishing the artifact a fail-closed security
# boundary then trusts is an OPERATOR action, not an agent's.
# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
# through the marker below and opens the pull request that moves this value. Nobody pushes
# this image from a workstation, and nobody pastes a digest here.
#
# OPERATOR, to complete the pin:
# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands
# 2. podman login gitea.home.keithjackson.dev
# 3. podman push --digestfile /tmp/conformance-runner.digest \
# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
# — the digest the REGISTRY returned, not the local manifest digest the build printed.
# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
#
# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
CONFORMANCE_RUNNER_IMAGE_DIGEST=""
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
# rendering the Job would fail visibly. It is now the same impossible all-zero digest
# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
# `sha256:` prefix this value carries — the trigger must strip it.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:63e1aa74289a8b77ef5cd2ec4e4d01e7fef638f965f2057329bc1991d3074f0d"
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
#
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
@@ -0,0 +1,104 @@
---
date: 2026-10-06
branch: feat/pm-agent-registry-pull-secret
title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
---
## What changed
The registry had two blockers. The first — the nodes resolving the registry hostname to the
non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
outright.**
`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
the committed worker Job template. It is a **second Secret holding the same Gitea token** as
`pm-agent-credentials`, because the two consumers need two formats: the application reads a
raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
duplication is the established pattern here (the Keycloak client secret is mirrored the same
way, for the same reason) and it is documented in both `.example` files.
The operator's part is the value, and one scope: the Gitea token must now carry
**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
indistinguishable from the Secret being absent.
**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
no container references, which is exactly why it is easy to leave unread — the same shape of
omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
needed because a `foreach` over an absent list yields no result.
The leverage is small and worth saying so: the image is digest-pinned, so a different pull
credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
and the rule costs four lines.
## What I verified
- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
pattern is satisfied by the first element, which is why the count is asserted separately),
`wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
template names exactly `pm-agent-registry` — so the template and the policy cannot drift
apart without the image build failing.
- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
`gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
`pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
refused it with both halves named — "has no top-level `sops:` block with a mac" and
"Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
earning its place: the file looked finished, and the value was a placeholder, so nothing
about reading the diff would have made it obvious. Encrypted, and the stage is green.
- **The registry really does refuse anonymous clients**, measured before building this
(2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
`{"message":"token is required"}`. That is why this is a credential and not a posture
tweak.
## What I did NOT verify
- **That the pull now works.** Nothing has been pushed to that registry and the committed
digest is still the all-zero placeholder, so neither half of the registry fix has been
exercised end to end. The first real pull is what proves both at once — and if it fails, the
error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
`unauthorized` is this credential.
- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
confirm the effect, because the dispatcher pod's retained events are all repeating
`BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
before the play ran. Not evidence either way.
- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
the documented read scope for packages; nothing here has exercised it, because no package
exists yet.
- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
`auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
authentication rather than be mistaken for working.
- **Anything about runtime.** Unchanged from the previous entries: the image has never been
built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
rules — has still never admitted or refused a real Job.
**Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
Three statements above are now out of date, and one was wrong when written:
- *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
branch now pins it.
- *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
- *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
org-package access is decided by account permission (team Packages unit, or org
visibility). Handled in the registry `.example`.
- **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
dispatcher's first call — behind the `user` scope category, so a token with only
`write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
The token needs `read:user` too.
@@ -0,0 +1,334 @@
---
date: 2026-10-18
branch: feat/keycloak-passkey-reconciler
title: "Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated"
---
## What changed
**Why this is a new branch and not an update to PR #72.** The brief for this work said to extend
PR #72 (`feat/keycloak-passkeys`) in place and not open a new PR. That premise was already stale
when I started: **PR #72 merged at 2026-10-18T05:44:04Z** (Gitea API: `state: closed`,
`merged: true`, `merge_commit_sha: 4d489f2`), and `git log origin/main` carries
`4d489f2 Merge pull request '…' (#72)`. A merged PR cannot be updated, and CLAUDE.md §1 leaves
exactly one path to `main`. So this is a fresh branch from `origin/main` in its own worktree, and a
new PR. For the same reason this is a **new** `decisions/` entry rather than an edit to
`decisions/2026-10-17-keycloak-passkeys.md`, which is now merged history describing a different unit
of work; it is the entry to read first for the passkey *design* decisions, which are not repeated
here.
**The problem.** PR #72 declared the WebAuthn passwordless policy and a restructured `browser-mfa`
flow in `platform/keycloak/keycloak-realm-import.yaml`. That declaration **could not take effect**.
A `KeycloakRealmImport` is a one-shot; prod's reads `Done=True / Started=False / HasErrors=False`,
so the operator will never re-run it and every realm-level field or flow declared after the first
import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it
changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the
2026-09-06 client-secret incident, third field over.
**What now makes it converge.** A **sibling** reconciler, modelled on
`platform/keycloak/client-secret-reconciler.yaml`:
- `platform/keycloak/passkey-reconciler.yaml` — ConfigMap (Python, stdlib only, **zero dollar
signs**) + CronJob on `*/15 * * * *`, `automountServiceAccountToken: false`, admin credential by
`secretKeyRef` only. Three passes, **all of which converge**:
1. **WebAuthn passwordless policy.** Strict read-modify-write of `GET`/`PUT /admin/realms/homelab`,
changing only the `webAuthnPolicyPasswordless*` keys that differ, readback-verified.
2. **`webauthn-register-passwordless` required action.** Strict read-modify-write of one
**literally-named** alias; converges `enabled: true` / `defaultAction: false` only.
3. **The browser authentication flow — BLUE/GREEN.** Build the declared tree under a new, unused
alias; verify it by readback; flip `browserFlow` in **one atomic PUT**.
- `ci/render-keycloak-passkey-policy.py` + the **generated**
`platform/keycloak/passkey-desired-state.yaml` — the desired state, rendered from the realm
import, `--check`ed by the merge gate.
- `conformance/tests/l0-static/keycloak-passkey-reconciler/` — `check.sh`, `chainsaw-test.yaml`
(`scope: whole-repo`), `mutations.py`.
- `platform-config/kustomization.yaml` — both new resources, in the **same** Kustomization as the
realm import.
- `platform/keycloak/README.md` — the layout table, the section heading, and the "does NOT reach the
live realm" section rewritten: all three declared things now converge, with the blue/green
mechanics, the rollback command and the `--no-bind` fallback written out.
### The flow design was reconsidered, and the first version of it was wrong
**This is worth recording because the superseded reasoning was not silly, it was just wrongly
scoped.** The first design of pass 3 was **audit-only**: compare the live browser flow against git,
report divergence loudly, and never write it — leaving the restructure as a one-time admin-console
action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow,
create-execution, set-requirement and bind-config as *separate* untransacted admin-API calls, so a
run dying between them leaves a **half-built flow bound as the login path** for every SSO-gated app
*and* kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the
conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every
platform-admin.
**That risk analysis was sound but it assumed the wrong shape.** It is true only if you mutate the
*bound* flow. The operator's instruction was *"if this can be done through GitOps, it should"*, and
audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already
forces. So:
- All the multi-call construction happens on a flow **nothing is using** (the declared alias plus a
hash of the desired shape, applied to *every* alias in the tree — reusing `browser-mfa-forms`
would mean writing *into* the bound tree). A failure at any point leaves an orphaned, unbound flow
and **zero user-visible effect**. The untransacted-calls problem dissolves.
- The built flow is **read back and verified** — nesting levels, ordering, authenticators,
requirements, and the conditional-OTP config *contents* — **before** anything is bound. The
regression the operator feared is now caught *before* it can bite rather than after.
- The flip is **one atomic PUT** of the realm setting `browserFlow`.
- **Rollback is one write.** The previous alias is printed in the run log, and there is **no
`DELETE` anywhere in the allow-list**, so the old flow survives as a working rollback target.
**The new safety property, enforced structurally rather than by intention.** Blue/green needs
flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone
cannot express "but not the flow that is bound", because its write entries are alias-parameterised.
So a second guard sits at the same choke point in `_http()`: the live `browserFlow` alias and its
**entire subtree** of sub-flow aliases and execution ids are read first, and every write is screened
against those sets before a socket opens. Protecting the whole subtree matters — adding an execution
to `browser-mfa-forms` changes live login behaviour just as surely as touching `browser-mfa`. It is
**fail-closed**: the sets start each run as "not determined", and while they are, *every* flow write
is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass
writes nothing.
**Idempotence became load-bearing rather than merely nice.** A rebuild-and-reflip every 15 minutes
would churn the login path on a schedule, so the bound flow is compared *structurally* and a match
produces **zero writes and no new flow**. Sub-flow and config *aliases* are deliberately not
compared: a flow this program built carries a hash suffix, so comparing labels would make it
permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is **reused**; a
half-built one is left alone and the next candidate alias is used, with the candidate count
**bounded** so a persistently failing build fails loudly instead of creating unbounded sprawl.
**Where the rollback target is recorded, and why not as an annotation.** The brief suggested
recording the previous `browserFlow` alias as an annotation on the CronJob or in a ConfigMap. The
pod **cannot**: it runs with `automountServiceAccountToken: false` and has no Kubernetes API access
at all, which is the same property that made the mounted-generated-ConfigMap design preferable to
reading the CR (and is why this change adds no `**/rbac.yaml`). Giving it write access to its own
annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse
bargain than the one it would buy.
So the record is in two places that cost nothing: the **run log** prints
`ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted.`, and — more
durably — **the realm itself**, because nothing is ever deleted. Every superseded flow is still
listed under `Authentication -> Flows`, so the rollback target is discoverable from the admin console
even if every Job log has rotated away. The README carries the rollback procedure.
**I achieved full convergence including the bind.** `--no-bind` exists as the deliberate fallback
(build and verify, leave the flip as a one-write human act) but it is not the default and was not
needed.
**No `**/rbac.yaml` was touched.** The brief floated having the reconciler read the
`KeycloakRealmImport` CR from the Kubernetes API, which would need a Role, a RoleBinding and a
ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state
**generated** from the import and **mounted**, so the pod keeps `automountServiceAccountToken:
false` and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with
`--check`, not by a second declaration being trusted.
**Sensitive paths touched** — checked against `infra/gitea/approval-policy.json` rather than from
memory, which caught one I had initially missed. **Three**:
| Path | Why it is in the policy | What I changed |
|---|---|---|
| `ci/**` | the pipeline itself | added `ci/render-keycloak-passkey-policy.py` |
| `conformance/**` | changing the tests changes what green *means* | added the L0 test directory |
| `platform-config/kustomization.yaml` | Flux wiring / which manifests are actually applied | added the two new resources |
**Not** touched: `**/rbac.yaml`, `clusters/**`, `platform/kyverno/**`, `.gitea/**`, `bootstrap/**`,
`infra/gitea/**`, `.sops.yaml`, any `*.sops.yaml`, `CLAUDE.md`.
So this PR **requires an approving review from someone who is not its author**, and
`ci/approval-policy.sh` is fail-closed, so it will block until that exists. I did **not** post
`/approve-override` — CLAUDE.md §1 names doing that on your own agent PR as exactly the
self-approval the stage exists to prevent. The operator needs to either review it or post the
override.
## What I verified
**Measured against the live cluster** (read-only `kubectl --context k3s-prod`):
- `keycloakrealmimport homelab`: `Done=True Started=False HasErrors=False` — the import is a
completed one-shot. **This is the premise of the whole change and it is measured, not assumed.**
- Keycloak image actually running: `quay.io/keycloak/keycloak:26.0.6`.
- `kustomization/platform-config` `.status.lastAppliedRevision` ==
`gitrepository/flux-system` artifact revision == `main@sha1:83d3568…`, which contains the PR #72
merge. So the inert declaration is genuinely applied and genuinely inert.
**`PUT /admin/realms/{realm}` does not touch users, groups, clients or roles in 26.0.6** —
established from source at tag `26.0.6`, not from documentation or memory:
- `RealmAdminResource.updateRealm` is the only handler. Its javadoc: *"Any user, roles or client
information in the representation will be ignored. This will only update top-level attributes of
the realm."*
- It delegates to `RepresentationToModel.updateRealm` →
`DefaultExportImportManager.updateRealm` (`model/storage-private/.../datastore/`), **lines
725–894**. I brace-matched that body and enumerated every `rep.getX()` call in it: it contains
**no** `getUsers()`, `getClients()`, `getGroups()`, `getRoles()`, `getRequiredActions()`,
`getAuthenticationFlows()` or `getAuthenticatorConfig()`. Those objects are **unreachable** from
the endpoint, not merely ignored by convention.
- Two findings that changed the design, both from the same read:
- `updateRealm` **does** read `rep.getBrowserFlow()`, so the endpoint can rebind the realm's login
path. The write therefore carries `browserFlow` straight back from the GET and a hard assertion
refuses the PUT if it would differ.
- The passwordless policy is applied **all-or-nothing and only when `RpEntityName` is non-empty**
(line 842), and `getWebAuthnPolicyPasswordless(rep)` substitutes a **Keycloak default** for every
null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and
one missing `RpEntityName` would drop the whole policy while returning 204. This is why the write
is a strict read-modify-write of the full live representation.
- `AuthenticationManagementResource.updateRequiredAction` (line 1205) is **per-alias** and likewise
**replaces** the provider from the representation (name, alias, priority, enabled, defaultAction,
config), and 404s when the alias is unregistered. Hence read-modify-write there too.
- `DefaultRequiredActions` registers `webauthn-register-passwordless` with `enabled=true`,
`defaultAction=false`, `priority=80` — confirming the realm import's claim that declaring it would
restate a default while deleting ten siblings.
- **The realm GET really does return what the read-modify-write needs.** This is load-bearing: if
the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset
them to defaults. `ModelToRepresentation.toRepresentation` (lines 471–482) sets **all ten**
`webAuthnPolicyPasswordless*` fields plus `ExtraOrigins`, and line 496 sets `browserFlow`. I
checked for an `internal`/`export` guard around that block — there is none in lines 380–500, so
those fields are emitted unconditionally, not only on the export path.
**The five flow-construction endpoints, read from 26.0.6 source before being used:**
- `POST /authentication/flows` — `AuthenticationManagementResource.createFlow`, rejects a duplicate
alias. Carries **no alias in its path** (the alias is in the body), which is why it needed its own
body-based screen; without that it escaped the protected-subtree guard entirely. My selftest
caught that, not my reading.
- `POST /flows/{alias}/executions/flow` — `addExecutionFlow`, body `{alias, type, description}`.
Creates the sub-flow **at requirement DISABLED**, and refuses if the parent `isBuiltIn()`.
- `POST /flows/{alias}/executions/execution` — `addExecutionToFlow`, body `{provider}`. Also
DISABLED unless the factory has exactly one requirement choice. Both assign `getNextPriority()`,
so **insertion order is the resulting order** — which is why the builder inserts depth-first in
declared order and then verifies by readback.
- `PUT /flows/{alias}/executions` — `updateExecutions`. **A trap I would have walked into:** it
compares `rep.getPriority()` against the stored priority, and a JSON body omitting `priority`
deserialises it as the int **0** — which would silently **reshuffle the whole execution order**
while only a requirement change was intended. Every requirement write therefore carries the
execution's current priority back. There is a mutation for it and a selftest assertion on the
resulting order.
- `POST /executions/{id}/config` — `newExecutionConfig`, creates an authenticator config and binds
it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and
the old flow already uses the declared alias.
**Offline proofs, all green:**
- The reconciler's `--selftest`, **extracted from the shipped ConfigMap** and run: **216 assertions
pass**. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the
all-or-nothing policy rebuild, the required-action replace, the `updateExecutions` priority trap,
and flow construction where everything is created DISABLED — so these are tested against the real
semantics rather than a forgiving fake.
- **A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds
and never touches the live flow.** Each case asserts the realm is still bound to the old alias,
that no write targeted the bound subtree, and that every live flow survives. This is the assertion
that makes blue/green a property rather than a claim.
- **Verify-before-bind is exercised, not assumed.** Two stub sabotages — a server that accepts the
requirement writes and applies the wrong value, and one that writes the conditional-OTP role under
`condition.evaluated.user.role` — both end with the flow **built but NOT bound** and the old flow
still live.
- `mutations.py`: **all 30 injected regressions caught**, including the six that are the blue/green
safety argument itself (build into the bound flow; never call the screen; reset the protected sets
to empty; bind before verify; report a failed build as success; reuse an existing flow without
verifying it).
- Mutation testing again found real gaps in my own work, which I fixed rather than weakening the
mutations. Across the two designs: no readback coverage on the realm write, no coverage of an
undeclared policy key surviving, a mis-indented anchor, and — in this design — **no coverage at
all of the bound flow being unreadable**, which is the fail-closed path. It also showed me that
**two of my proposed mutations were not real regressions**: the module-level initial value of the
protected sets is dead code (`run()` resets them first), and removing the early return after a
build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that
*are* load-bearing rather than pretending the originals had been caught.
- The new L0 `check.sh`: rc 0, all assertions. It asserts the renderer `--check`; that the realm
import and the desired state are rendered by the **same** Flux Kustomization; that the renderer's
and reconciler's writable key tuples are identical (parsed out of both sources with `ast`); that
`defaultAction` is `false`; that the realm import declares **no** `requiredActions` block
(previously only a comment saying "DO NOT ADD ONE"); that the **sibling** reconciler's allow-list
was not widened; that **nothing is deletable** and the execution-/config-id-scoped write endpoints
stay off the list (they carry no flow alias, so they could not be screened); and — the
load-bearing one — that the **protected-subtree guard exists in the source, is wired into
`_http()`, resets to "undetermined" every run, screens the body-only create-flow endpoint, and
that the policy write still refuses to rebind `browserFlow`**.
- `ci/render-keycloak-passkey-policy.py --write` output round-trips: the generated JSON is accepted
by the reconciler's `declared()`, its 10 policy keys equal the reconciler's `POLICY_FIELDS`, and
it flattens to the exact 11-step tree the realm import declares, with
`conditional-otp-platform-admin` bound at level 4.
- `${BASE_DOMAIN}` is carried into the generated ConfigMap **verbatim**, and both files sit in
`platform-config`, so one substitution pass decides the RP ID.
- Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed
manifest.
**The conditional-OTP regression is detectable — and now also preventable.** The comparison is
key-by-key on the authenticator config contents, and because it runs **before** the bind, a flow
carrying the bug never becomes the live login path. `--selftest` covers it three ways: the historical
`condition.evaluated.user.role` key injected into a *built* flow (built, verified, **not bound**,
old flow still live); the same bug already live in the *bound* flow (reported, then **fixed** by
building and binding a correct flow); and the config disappearing entirely. `mutations.py` deletes
the comparison and the undeclared-key report and requires the selftest to notice both. The suite
also covers a requirement flipped from `ALTERNATIVE` to `REQUIRED`, the un-restructured legacy flow,
a correct-but-unbound flow, and a cyclic declaration.
**Gate:** see the gate section of the PR / the run quoted in my report — rc and per-stage results
are recorded there rather than paraphrased here.
## What I did NOT verify
- **The live realm's actual passkey state is entirely UNVERIFIED.** I never authenticated to the
Keycloak admin API. Doing so needs the `keycloak-admin-credentials` password, which CLAUDE.md §6
forbids decrypting or printing, so I did not read the realm's current
`webAuthnPolicyPasswordless*` values, its required-action list, or its `browser-mfa` executions. I
therefore **do not know** how much drift the first real run will report, nor that the live flow is
in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift
the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
- **Nothing has ever run against a real Keycloak.** Every behavioural proof is against an
in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This
matters more for the blue/green flow builder than for anything else in the change, because it is
**five endpoints I have never exercised**. Specifically unproven in reality:
- That `POST /flows/{alias}/executions/flow` and `.../executions/execution` accept the bodies I
send and assign priorities in insertion order. If they do not, the build produces a tree that
fails verification — which is the safe failure (nothing is bound, login unaffected), but it
means **the flow would never converge** and the Job would fail every 15 minutes.
- That `PUT /admin/realms/homelab` accepts a full round-tripped representation. The analogous
*client* endpoint rejects one with `400 "Cannot parse the JSON"` (measured 2026-09-06), and the
sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm
endpoint differs; no measurement here confirms it. **If it does not, both the policy pass and
the atomic bind fail closed** — loudly, writing nothing.
- The exact JSON shapes compared: `AcceptableAaguids` as `[]` rather than absent or null,
`CreateTimeout` as a JSON int, `RequireResidentKey` as the literal string `"Yes"`. A shape I did
not anticipate would show up as **perpetual drift** — a write every 15 minutes that never
converges — rather than as an error.
- That `authenticationConfig` on an execution-info response is the config **id** (I read it from
the representation class, but never saw a live response).
- **I did not verify the alias-collision behaviour against a real realm.** The recovery logic
assumes `POST /flows` returns a non-2xx on a duplicate alias (the source checks
`getFlowByAlias != null` and raises `ErrorResponse.exists`), and that a config alias collision
likewise fails. If either silently succeeded, the recovery path would behave differently than
modelled.
- **Flow sprawl is bounded but real.** Every genuinely new desired shape creates a new flow and
**nothing is ever deleted** — by design, because the old flow is the rollback target. Over many
shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented
beyond "an operator deletes them". I consider that the right trade (deletion is the one operation
that could destroy a rollback target) but it is a cost, not a non-issue.
- **No L2 behavioural test was added.** The reconciler has an `--audit` mode built for one and the
sibling has `tests/l2-behavioral/keycloak-client-secrets-match-git`, but I added no equivalent.
And `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 would not run in CI anyway. **CI proves the
offline subset only; it proves nothing about the live realm.**
- **No alerting.** Unlike the first design there is no longer any failure that is "expected by
design", so a PrometheusRule on `kube_job_status_failed` would now be meaningful — and I did not
add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret
key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
- **That passkey login works.** No passkey has been enrolled and no browser has been pointed at
this. Nothing offline can prove a WebAuthn ceremony completes, that the `ALTERNATIVE` fork
degrades to the password form via `CREDENTIAL_SETUP_REQUIRED`, or that a platform-admin is still
asked for TOTP on the password branch. The README's four-step manual check remains the only thing
that would establish it, and it remains undone.
- **That the RP ID is right for the origin.** `${BASE_DOMAIN}` is a legal registrable-domain suffix
of `keycloak.${BASE_DOMAIN}` on paper, but a wrong RP ID fails **in the browser**, before Keycloak
is involved, with no server-side log line. The reconciler will happily converge the realm on a
wrong RP ID; it checks that git and the realm agree, never that git is correct.
- **The first real run will change how everyone logs in.** Once this merges and Flux applies it, the
next tick builds a flow and — if it verifies — **flips the realm's browser flow**. The UX change
is deliberate and documented (username screen, then password), but it is the first time it will
have happened, unattended, on a schedule. The rollback is one write and the old flow survives;
that is the mitigation, and it has not been rehearsed either.
- **CI's own result.** My gate run is local. `gitea-settings` cannot run locally, and
`ci/approval-policy.sh` reports green N/A offline while this branch touches three sensitive paths,
so **the PR is guaranteed to need a non-author approving review** and I have not seen CI report. I
did not and will not post `/approve-override` on my own PR.
- I re-measured branch protection this session (`protected: true`, `user_can_push: false`,
`enable_status_check: true`, context `gate / gate (pull_request)`, `required_approvals: 0`), but
the branches endpoint does **not** expose `block_admin_merge_override`, so that row of CLAUDE.md
§2 is **not re-confirmed** by me.
+82
View File
@@ -0,0 +1,82 @@
---
date: 2026-10-18
branch: fix/pihole-local-ttl
title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
---
## What changed
The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
**asymmetry in cacheability** that we created.
Measured on prod 2026-10-18:
```
dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
```
dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
rather than forwarding live. So:
* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
lookup, which is maximum exposure to any momentary gap;
* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
(standard forwarder behaviour when the forward target does not answer) it latches the public WAN
address and serves it to **every LAN client** for five minutes.
So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
regenerated with the canonical render command.
## What I verified
* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
143 (decrementing), which a pure forwarder could not do.
* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
address), so a fallback always has a wrong answer available.
* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
`/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
`FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
--no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
`apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
## What I did NOT verify
* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
nor the router caching the private answer, nor the flap stopping. **Verify after merge with
`dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
— so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
public answer latches again. That is a real availability problem, not a caching one.
* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
(`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
(tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
that decision, and it is deliberately **not** in this change.
* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
there. Only this one workstation's resolver chain was inspected.
* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
`home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
next recurrence would again be found by a human noticing. That gap is not closed here.
@@ -0,0 +1,78 @@
---
date: 2026-10-18
branch: fix/renovate-cron-window
title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
---
## What changed
`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
never have produced a single PR.**
Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
re-checks it and declines to open PRs outside it. So:
```
cron 03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local
inside? -> NO (3h45m before the window opens)
```
The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
green tick forever, indistinguishable from "nothing needed updating". That is the same
"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
pending updates PR #73 was written to surface.
Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
position moves:
| | local window | in UTC |
|---|---|---|
| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
A cron cannot follow DST, so this is the only way to get it right once.
## What I verified
* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
`America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
both inside `03:00–06:00`. Computed, not reasoned.
* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
`timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
when PR #73 was authored.
* The edited workflow parses as YAML.
## What I did NOT verify
* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
usable PR.
* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
`customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
change it**, because guessing at a config key that could fail validation and take the whole run
down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
validation warning naming `regex`.
* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
schedule". That makes the first dispatch's output less informative than it looks; to see what
Renovate *would* do, either dispatch inside the local window or override the schedule for that
run. Not tested either way.
* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
does.
* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
return. A static check comparing the two is feasible and is not written.
@@ -0,0 +1,78 @@
---
date: 2026-10-18
branch: fix/renovate-invalid-config
title: "renovate.json was invalid, so the first manual run errored out"
---
## What changed
The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
**`renovate.json` was never a valid Renovate config.**
It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
top-level options**, so the entire config failed validation. Proven with the official validator
rather than inferred:
```
$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
ERROR: Found errors in configuration
Invalid configuration option: _comment
Invalid configuration option: _docker_comment
Invalid configuration option: _ignorePaths_comment
Invalid configuration option: _kubernetes_comment
Invalid configuration option: _unmanageable_comment
```
This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
happens"; #73 shipped both.
**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
is preserved as comments positioned next to what it explains, which a flattened `description`
string would have lost.
A header on the new file records the failure, the validator command, and the instruction to run it
before committing — this class of bug is invisible to the repo's own gate.
## What I verified
* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
from the workflow.
* **The fix validates**: the same validator against `renovate.json5` returns
`INFO: Config validated successfully against 1 file(s)`.
* **Nothing was dropped in the conversion**, checked programmatically against `git show
HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
prints a normalisation diff that mentions it, which is its own migration suggestion, not a
removal.
* **74 comment lines survive** in the new file, so the rationale is intact.
* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
the workflow's fail-hard preflight is satisfied.
* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
`custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
validator accepts `regex`, so that concern was unfounded and nothing here changes it.
* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
conformance/` returns only comment prose.
## What I did NOT verify
* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
bot works — only that these two specific defects are fixed.
* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
`dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
`prHourlyLimit: 2` / `prConcurrentLimit: 5`.
* **Whether the managers actually extract anything.** A valid config that matches no files is still
a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
`managerFilePatterns` are right, and I have not seen them.
* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
discovery is long-standing and the validator accepted the file, but the workflow's image tag was
not cross-checked against the version that added it.
* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
here, and `conformance/**` is a sensitive path.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
+374
View File
@@ -0,0 +1,374 @@
---
date: 2026-10-29
branch: feat/image-pipeline
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
---
## What changed
Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
both `build-image.sh` scripts said "publishing is an operator action".
### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
versions' source:
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
Giving jobs the DinD daemon would have made every branch push root on the node, because that
daemon is `privileged: true`.
**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
valid_volumes) is **untouched**.
- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
business sitting next to a privileged container. Flux reconciles it as the new Kustomization
`buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
- **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
digest. The securityContext, every line measured (see *What I verified*):
- `privileged: false`;
- uid/gid 1000, `runAsNonRoot`;
- `capabilities: drop ALL, add SETUID, SETGID`;
- `allowPrivilegeEscalation: true`;
- `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
- `readOnlyRootFilesystem: true`, with five emptyDirs.
Other settings:
- `--oci-worker-no-process-sandbox`;
- no ServiceAccount token;
- memory limit 4Gi, which covers build steps too;
- cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
(ConfigMap).
- **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
client against that CA.
- **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
- **NetworkPolicies:**
- default-deny;
- ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
`app.kubernetes.io/name=act-runner`);
- egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
registry push.
- `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
`checksums.txt`, plus the address and certificate name CI dials.
- **How DinD job containers appear to Cilium (measured, read-only).**
`iptables -t nat -S POSTROUTING` in the dind container shows
`-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
another, or a job from act_runner itself. Per-job authentication is the client certificate.
- **Admission: no Kyverno exception needed.**
- Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
`conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
`require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
`restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
namespace in the job-shape policies' case. `buildkit` carries no such label.
- There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
- The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
`Unconfined` seccomp and AppArmor.
- `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
its shape (both directions) satisfies it.
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
- `kernel.unprivileged_userns_clone = 1`;
- `user.max_user_namespaces = 127206`;
- no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
- AppArmor enabled (`Y`).
So this is a satisfied prerequisite, not an open one.
- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
changes what green means; flagged for the reviewer).
- The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
issuer must be ACME". That is broader than the spec it serves:
- R-TLS-01 and docs/08 concern *externally reachable* services;
- ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
- a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
- The new clause (5b) exempts a Certificate annotated
`homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
- no Gateway listener serves its Secret;
- every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
- it has no IP, URI or email SANs;
- its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
- a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
- Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
- **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
check with a specific message.
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
`tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
`BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
- the cert verifies against the CA;
- the cert is a client-auth cert;
- the key matches the cert.
It never prints a value. `--check` reports whether the stored copy is older than the current
certificate.
### Verification moved into the Containerfiles
`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
pushed.
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
on PATH, and five tools report their pinned versions.
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
`[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
are real assertions now.
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
header):
- **uid:** exact.
- **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
TCP connection works. A negative control without `--network=none` fails that check.
- **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
It is approximated by permission:
- `touch /should-fail` must fail;
- a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
`/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
to `/var/tmp` tripped it.
The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
enforce.
- **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
is exactly what the pod's emptyDir replaces.
The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
- **`build-image.sh` (all three):**
- verify is `build --target verify` and the image is `build --target <image>`;
- a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
arg list therefore exists once;
- the local podman path still works.
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
the toolbox's build-arg list.
### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
creates a `remote` builder, and runs `buildx inspect --bootstrap`.
- **It fails loudly, never skips.** It stops with a clear message if:
- the secrets are absent;
- buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
applies it only after merge), stale certificate, NetworkPolicy.
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
because they must build.
- agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
context. That toolbox exists in no registry, so the toolbox is built as its base even when
its own inputs did not change.
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
- The digest is read back from the registry and compared with the client's metadata.
- agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
`AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
- `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
Renovate and the retention script assume.
- **Unchanged from the first iteration:** change detection, publish idempotence,
`REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
DinD prune step is gone, because nothing is built in DinD now.
### Unchanged from the first iteration (still true)
- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
- `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
until the first publish;
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
The admission policies are untouched.
- **`renovate.json5`**:
- the `custom.regex` marker manager;
- `enabledManagers` `regex` → `custom.regex`;
- the `kubernetes` manager disabled for these three repos;
- `ignorePaths` narrowed so `conformance/versions.env` is visible.
`renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
using `RENOVATE_TOKEN`, which now needs `read:package`.
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
`conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
plan, conformance README, devcontainer README, workflows README). The docs are updated again
for BuildKit, and there is a new `platform/buildkit/README.md`.
### Decisions and pushback
- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
Checked against Gitea **v1.24.3** source:
- `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
`gitea-actions` (ID −2) and sets no token scope;
- `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
401. The Actions token cannot even *pull*;
- `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
and no branch for that user. Even past login it would get at most read on a public org;
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
pattern. With BuildKit the credential reaches the daemon through the buildx session (a
0600 `config.json` in the job's temp dir), never through the cluster.
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
`custom.regex`;
- that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
`migrateAndValidate`), not only in the validator;
- `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
- `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
the workflow actually runs.**
- `renovatebot/[email protected]` defaults `renovate-version` to `39`
(`action.yml`, `src/docker.ts`), which is 39.264.0.
- The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
44.167.0.
- **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
`ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
`fileMatch` was renamed in 40.2.0.
- So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
- Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
The only `--strict` complaint is a pending migration of the pre-existing top-level
`"docker": {"enabled": true}` block, which I left alone.
- **Unworkable as briefed: tracking the worker Job template's digest.**
- `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
forever.
- `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
(`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
deliberately untracked.
- For the same loop reason, `conformance/versions.env` is **not** an input of the
conformance-runner build, and a pipeline-only change does not republish on main.
- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
- Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
it lacks.
- A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
version it was pushed under (`routers/api/packages/container/manifest.go`
`createPackageAndVersion`), and resolves a by-digest pull from those files
(`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
script keeps every digest git pins.
- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
`digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
sensitive path, so it waits for an administrator's `/approve-override` every time.
## What I verified
**Proved locally, against the exact pinned images:**
- **Rootless buildkitd, configured as the manifest configures it.**
- Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
`--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
- The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
still ran.
- It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
GC config applied (max 4.29 GB, reserved 1.07 GB).
- **The securityContext, item by item.** Each requirement was measured by taking it away:
- **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
- **no `SETUID`/`SETGID`:** rootlesskit dies with
`newuidmap … operation not permitted` → both capabilities are required;
- **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
probe client's `~/.docker`, which is now an emptyDir;
- **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
cache. rc 0, 1m32s total:
- `TOOLBOX VERIFIED` (42 s);
- `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
FROM the in-run toolbox through the `target:` context, with the log showing
`[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
- `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
- every stage printed `confirmed: no network`.
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
- no client cert → `remote error: tls: certificate required`;
- a client cert from another CA → rejected the same way;
- the valid cert → accepted.
Through the driver:
- another CA's client cert → rc 1;
- the wrong CA for the server → rc 1;
- an unreachable host → rc 1 in about 20 s, with the explanatory message;
- missing secrets → rc 1, naming the three secrets.
- **The local podman path**, `build-image.sh` for all three (rc 0).
- Negative control: a verify stage expecting a different promtool version printed
`DRIFT promtool want 9.9.9` and failed the build.
- Negative control: the `--network=none` check fails when the flag is removed.
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
- a good client cert;
- a server cert (not client-auth);
- a mismatched key;
- a cert from another CA.
**Measured live, read-only:**
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
- DinD's MASQUERADE rules;
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
ClusterIssuers;
- the act-runner pod labels.
**Renovate (unchanged since the first iteration):**
- the validator with 44.167.0: exit 0;
- `extract` finds all four markers;
- the `kubernetes` duplicates are `skipReason: disabled`;
- 39.264.0 rejects `main`'s config.
**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
the tree being gated.
## What I did NOT verify
- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
`kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
and is expected to fail at the builder preflight**, which is the designed loud failure. The
following are therefore all unproven in-cluster:
- kubelet admitting the pod;
- containerd honouring `appArmorProfile: Unconfined`;
- the five emptyDirs;
- cert-manager issuing the three certs;
- the CiliumNetworkPolicies.
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
is the normal case, but it is the cluster's first test.
- **The NetworkPolicies are untested.** In particular:
- that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
reasoning, but it is unmeasured;
- that world :443/:80 covers every download the builds make. A dnf mirror on another port
would fail a build visibly.
- **No push, anywhere.** The whole publish path is unexercised:
- `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
- `push=true`;
- the metadata digest compared with the registry read-back;
- agent-runner FROM `toolbox@sha256:`;
- the immutable-tag guard;
- idempotence.
So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
with repositories) absorbing ~2 GB per toolbox publish until retention runs.
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
had 7–9 GB free. It is untested under concurrent pressure.
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
buildkitd only reads it at start; the client copy needs the script re-run.
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
PR job, can run build steps.
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
`read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
untested.
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
registry.
@@ -0,0 +1,172 @@
---
date: 2026-10-29
branch: feat/keycloak-passkey-reconcile
title: "Passkey reconciler review fixes: bind only on git's say-so (rollback sticks), protect the real 26.0.6 sub-flows, screen the requirement PUT by body id, alert on failure"
---
## What changed
This branch carries the unmerged passkey reconciler (commit `36fcb7b`, cherry-picked with its
authorship, recorded in `decisions/2026-10-18-keycloak-passkey-reconciler.md`) plus the fixes for a
review of it. Read that entry for the design; this one records what the review found and what
changed.
**1. HIGH: the bind was automatic, and rollback did not stick.** Within one tick of merge the
reconciler would build and BIND `browser-mfa-<hash>`, a login flow no human had tried. Worse, the
README's rollback (rebind `browser-mfa`) was undone every 15 minutes: the next run found the
verified candidate again and rebound it, and a hotfix inside the bound flow was "corrected" by
building and binding an `-r2`.
- New git-declared switch: `bind-browser-flow` in `platform/keycloak/passkey-desired-state.yaml`,
a second key of the same ConfigMap, so it reaches the pod through the existing mount. **Default
`"false"`.** It is the one hand-edited line in a generated file: the renderer carries it through
on `--write`, and `--check` (run by the L0 gate) refuses anything but the quoted strings `"true"`
/ `"false"`. An unquoted `true` is a YAML boolean, which a ConfigMap rejects at apply time and
which would have failed the whole `platform-config` Kustomization, realm import included.
- `"false"`: pass 3 builds and verifies the candidate (or reuses a verified one), logs
`WOULD BIND <alias>`, exits 0 with `KEYCLOAK_PASSKEY_RECONCILED_OK_PENDING_BIND`, and never PUTs
a changed `browserFlow`. That holds after an earlier bind too, so the operator's rebind sticks.
If the bound flow *is* the candidate alias and has been hand-edited, it is left alone (no `-r2`
is built beside it).
- `"true"`: binds as before, and keeps asserting it. That is GitOps working; the README says so,
and says not to hotfix the flow while it is `"true"`.
- The CronJob passes no flag. `--no-bind` still exists for a hand-run Job and can only narrow
git's value. The L0 check asserts the CronJob carries no `--` argument.
- Rollback is now: a PR setting `"false"` (first), then one rebind write. README documents the
order and why it matters, plus how to test the candidate on one client (a per-client
`account-console` browser-flow override) before the go-live PR.
**2. HIGH: the protected-subtree guard read the wrong field for sub-flows.** Read from source at tag
26.0.6 (`services/src/main/java/org/keycloak/services/resources/admin/AuthenticationManagementResource.java`,
fetched from GitHub this session): `recurseExecutions` sets `displayName = flowRef.getAlias()` and
`flowId` on a sub-flow row and never calls `setAlias`. `alias` is set only on authenticator rows,
where it holds the bound **config's** alias. So the protected set was `{bound}` alone, and a write
into `browser-mfa-forms` would not have been refused. The old stub returned `alias` on sub-flow
rows, which is why all 216 assertions passed anyway.
- New `subflow_alias()` reads `displayName` and cites the source. I did not resolve by `flowId`:
`getFlows()` (same file) filters to `isTopLevel()`, so a nested sub-flow's id resolves to nothing
through the flows list. A sub-flow row with no `displayName` makes the listing unreadable, which
leaves the flow pass fail-closed.
- The stub now returns rows shaped like 26.0.6's: sub-flow rows carry `displayName`, `flowId` and no
`alias`/`providerId`. Authenticator rows carry a display *type* as `displayName` (not the
provider id) and the config alias as `alias`. Nulls are omitted.
- New assertions: after a real run, `browser-mfa-forms` and `browser-mfa-conditional-otp` are in
the protected set, and an execution add or requirement write into them is refused. A missing
`displayName` fails closed.
**3. MEDIUM: `PUT /flows/{alias}/executions` was screened on the wrong field.** `updateExecutions`
resolves `realm.getAuthenticationExecutionById(rep.getId())` without checking membership in
`{alias}`. The same handler also renames the flow named by `rep.getFlowId()` when the body carries
`authenticationFlow` + `displayName`. Now:
- `body["id"]` is screened against the protected execution ids.
- It must also be in `_BUILD_EXECUTION_IDS`: the ids read back from the flow this call just
created, open only inside `build_flow`.
- The config `POST /executions/{id}/config` is held to the same build scope.
- The body may carry only `id`, `requirement` and `priority`.
- Selftest cases cover each of these.
**4. Stale docs on sensitive paths** now describe what ships, including the switch:
`platform-config/kustomization.yaml`, the chainsaw test header, the desired-state header and its
renderer template, the renderer's own header and `--check` message, and the README.
**5.**
- `backoffLimit: 0`: every failure is deterministic within a tick, and a retried build burns
another of the 5 candidate aliases.
- New alert `KeycloakPasskeyReconcilerFailing` in `platform/monitoring/alerts.yaml` (warning,
`for: 15m`, `kube_job_status_failed` on `job_name=~"keycloak-passkey-reconciler-.*"`), the same
shape as the sibling's. It has promtool tests in
`platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`, in both directions plus
"the sibling's failure is not this alert" and "one transient failure stays quiet". The alert
ratchet in `alert-rules-unit-tested` would have refused it untested. A new runbook,
`runbooks/keycloak-passkey-reconciler.md`, is projected by the runbooks generator.
- The stale "Renovate does not manage plain `image:` tags" comment is corrected: `renovate.json5`
enables the `kubernetes` manager over `platform/**`. The image is pinned to
`python:3.13.15-alpine3.23@sha256:6438599575cca0d1df94aeee0d2ae088d4d8846eab554b2ee7784a3a6df0d516`,
the multi-arch index digest read from Docker Hub today. The tag is mutable: the sibling
reconciler's pod on prod reports `imageID ...@sha256:75f27d68...` under the same tag, which is
neither today's index nor today's amd64 manifest.
**6. Hygiene.** The ~160-line header that appeared in both the YAML and the script is now kept once,
in the script, where it ships next to the code and its selftest. A short header above the
ConfigMap points to it. Nothing else was restructured, apart from the build's second half moving
into `_finish_build()` so the build scope can be opened and closed around it.
Sensitive paths touched: `ci/**` (renderer), `conformance/**` (L0 check, mutations, chainsaw),
`platform-config/kustomization.yaml` (comment only). This needs the operator's
`/approve-override` or a non-author review. I did not post one.
## What I verified
- **Local toolbox gate, rc 3** (pinned toolbox via `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE
shim bind-mounting the main `.git`, run on the committed branch). Stages: `spec-immutability`
pass (the spec repo resolved as a sibling), `decisions-log` pass, `lint` pass, `l0` pass
(including `keycloak-passkey-reconciler`, `alert-rules-unit-tested` and
`alert-runbooks-resolvable`), `render-golden` pass, `l1` pass, `approval-policy` pass (N/A
offline; it is guaranteed to fail in CI). **`gitea-settings` did not run** (no
`SETTINGS_API_TOKEN` locally) and is the only stage that did not run. L2/L3 `conformance` is
skipped by design (no cluster). rc 3 is **not** green.
- `platform/monitoring/tests/run.sh platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`
in the toolbox: `promtool check rules` SUCCESS (54 rules), `promtool test rules` SUCCESS,
promtool 3.14.0.
- **Selftest: 259 assertions pass** (was 216), extracted from the shipped ConfigMap by the L0
check.
- **`mutations.py`: 40 of 40 injected regressions caught** (was 30). The ten new ones each undo
one fix:
- sub-flows read by `alias` again;
- an unnameable sub-flow tolerated;
- the requirement PUT screened on its path only;
- execution writes not held to the build tree;
- extra body keys allowed;
- the bind switch ignored in `run()`;
- the bind switch ignored in `reconcile_flow()`;
- an absent switch reading as true;
- an invalid switch guessed at;
- `-r2` built beside a hand-edited bound flow.
"Sub-flows by `alias`" is caught less cleanly (the selftest aborts on an unreadable listing
rather than failing an assertion). It is still caught.
- The renderer accepts `"true"` and `"false"` and refuses `true` (unquoted) and `"yes"`. I
checked this by hand-editing the committed file and running `--check`.
- The 26.0.6 source for `recurseExecutions`, `updateExecutions` and `getFlows` was read directly
(lines ~225-231, 658-724 and 747-806 of the file at that tag).
- Read-only `kubectl --context k3s-prod`: Keycloak pods run `quay.io/keycloak/keycloak:26.0.6`
(2 replicas). The only CronJob in `keycloak` is `keycloak-client-secret-reconciler`, so the
passkey reconciler is **not** live.
## What I did NOT verify
- **Nothing has run against a real Keycloak.** The stub's row shapes now follow the 26.0.6
source, but they are still my reading of that source, not a captured response. In particular,
I have never seen a live executions listing to confirm `displayName` on a sub-flow row. If it
differs, the failure is fail-closed (no build), not fail-open.
- **The candidate flow has never been tested in a browser.** That is now a precondition of the
go-live PR (README, "Testing the candidate flow before binding it"). Nobody has run those steps.
I also have not checked that the per-client `account-console` override behaves as described on
26.0.6.
- **"Merging changes nothing users see" is true of login, but passes 1 and 2 still write the
realm.** They set the passwordless policy and enable the enrolment action. That action is
enabled by default in 26.0.6, but I did not read the live realm, because no admin API calls were
allowed. Pass 3 also creates the unbound candidate flow (realm DB writes, invisible to login).
- **Race in pass 1, not fixed:** the policy PUT is a read-modify-write that carries `browserFlow`
back from its own GET. An operator rebind landing between that GET and PUT would be reverted.
The window is one HTTP round trip and only exists when the policy has drifted.
- **The alert has never seen a real series.** The CronJob has never run, so
`kube_job_status_failed{job_name=~"keycloak-passkey-reconciler-.*"}` does not exist yet. Its
shape is copied from the sibling's rule, which does have live series.
- **The digest is of today's index.** I did not pull it or run the image. Renovate's handling of a
`tag@digest` pin under the `kubernetes` manager is from its documentation, not from a Renovate
run here.
- **CI's own result** is in the PR, not here. `approval-policy` is expected to fail until the
operator overrides.
### Follow-ups (out of scope here, deliberately not done)
- A dedicated least-privilege service-account client for the reconciler, instead of the
master-realm break-glass admin it shares with the client-secret reconciler.
- A NetworkPolicy for the `keycloak` namespace.
- Whether a passkey login should skip TOTP for platform-admins. This is the PR #72 design
decision, currently "no TOTP after a passkey".
- A `KeycloakPasskeyReconcilerStale` rule (no successful run in N hours), like the sibling's.
Only the failure alert was in scope.
@@ -0,0 +1,57 @@
---
date: 2026-10-29
branch: fix/renovate-config-filename
title: Renovate has failed every run since #79 — the workflow still named renovate.json
---
## What changed
**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
322 (task 513, schedule, 08:15 UTC):
Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
repository Actions secret (listed by name via the API; value not read), so the token preflight
passes and this filename is the only blocker observed.
**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
it, and the gate stayed green through 11 days of failures.
**Second blocker, same PR:** the workflow did not pin a Renovate version, so
`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
## What I verified
- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
no `renovate.json` (`git ls-tree`).
- The new guard, both ways, run standalone: on this branch it prints
`workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
also matched its own explanatory comment in the workflow and failed. That is why the grep is
anchored to non-comment lines.
- Toolbox gate: result and rc are reported in the PR, not predicted here.
## What I did NOT verify
- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
action's default (`'39'`). The validation results are the PR #81 worker's.
- **Renovate has still never completed a run.** This removes the observed blocker. Others may
follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
Nothing notices a scheduled workflow that is red every day. That gap stays open.
- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
reports that stage as N/A.
@@ -0,0 +1,81 @@
---
date: 2026-10-30
branch: fix/gitea-admit-buildkit
title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
---
## What changed
**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
open the direct ClusterIP path from build code to Gitea, which nothing uses.
### Why: the brief's hypothesis was half right
Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
namespaces either.
## What I verified
All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
Gateway reach Gitea and get logged. This one never arrived.
- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
(Gateway VIP `192.168.2.20`) gave:
- from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
- from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
No token was involved, so this cannot be a package-permission problem. The bot account and the
`REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
`buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
the 403 itself in 0 ms.
- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
(`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
`default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
(1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
(391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
happens on the source's egress, not on the destination's ingress.
- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
is no Gitea identity.
- The edited YAML parses (three documents, names unchanged).
## What I did NOT verify
- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
"Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
same pod spec and the same policy, but it is not the original request.
- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
did not read them.
- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
for a first push that creates a new package. These only become testable after this lands.
@@ -0,0 +1,96 @@
---
date: 2026-10-30
branch: feat/pin-pihole-jackett-plex
title: Pin pihole, jackett and plex to the digests already running, and make Renovate able to follow them
---
## What changed
- **`catalog/{pihole,jackett,plex}.yaml` `source.image`** pinned to the tag whose digest was
already running, so the *image* does not change:
- `pihole/pihole` (no tag) -> `pihole/pihole:2026.10.0`
- `lscr.io/linuxserver/jackett:latest` -> `lscr.io/linuxserver/jackett:v0.24.2865-ls7`
- `plexinc/pms-docker:latest` -> `plexinc/pms-docker:1.43.4.10903-e5521bd8c`
- **`apps/wave-0/{pihole,jackett,plex}.yaml`** re-rendered (`render.py --all catalog -o apps
--wave-layout --no-image-automation`, in the toolbox). The diff is exactly the three app-container
`image:` lines; nothing else moved.
- **`catalog/jackett.yaml`** `update.range` comment: "stay on the Jackett 0.22.x stable line" was
false (0.24 runs, and `>=0.22.0` has no ceiling). Range unchanged; comment corrected.
- **`renovate.json5`**, four packageRules plus a comment block:
- jackett `versioning: regex:^v(major).(minor).(patch)-ls(build)$` (Renovate's linuxserver example).
- jackett **`overridePackageName: ghcr.io/linuxserver/jackett`**: not in the brief, and needed.
Renovate's docker datasource reads at most `dockerMaxPages` (20, a global-only option) tag
pages from any host except a short list that includes ghcr.io (1000 pages). lscr.io is not on
that list. Jackett has **32,093** tags, oldest first, ~1000 per page; the running tag is #31,980.
Before this rule the dry run extracted jackett at `v0.24.2865-ls7` and **proposed nothing**,
with no warning. The override changes only the lookup; the manifests keep `lscr.io/...`.
Every jackett rule matches on `depName`, because the override rewrites `packageName`.
- jackett `schedule: after 3am and before 6am on sunday`, i.e. weekly. Measured: 13 ls builds in
12 days (ls7 built 2026-10-17, ls20 built 2026-10-29). Each is a `patch` and auto-merges, so on
the daily window it would mean a PR, a gate run and a jackett restart (Recreate) every day.
Weekly picks the newest tag, so it skips nothing. `catalog/jackett.yaml update.cadence` still
says daily. That field fed the retired Flux image-automation, and the rule's description
says the rule is now the cadence that takes effect.
- plex `versioning: regex:^(major).(minor).(patch).(build)-[0-9a-f]+$`.
- pihole: **no rule**, as the operator decided. CalVer parses under default docker versioning.
Patches and minors auto-merge, and a year rollover is a major (dashboard approval). This is
written in a comment.
- Corrected the "WHAT RENOVATE STILL CANNOT FIX" comment. It listed these three as floating;
they publish immutable version tags.
- No `allowedVersions` mirror was added. None of the three catalog ranges has a ceiling
(`>=2024.0.0`, `>=0.22.0`, `>=1.40.0`), so #86's mirror has nothing to encode for them.
- **Runbooks:** `image-policy-stalled.md` no longer says jackett and pms-docker are
`latest`-only. In `plex-quicksync.md`, "the floating-image caveat" became "the image-version
caveat", because the deployment no longer runs `:latest` with an implicit `Always`, and the symptom
is now "after a pms-docker bump". `plex-claim.md`'s two "running `:latest`" mentions are
now in the past tense.
## What I verified
- **Live digests** (read-only `kubectl --context k3s-prod`, pod `.status.containerStatuses[].imageID`):
pihole-0 and pihole-1 `sha256:d34134716fc3…`, jackett `sha256:4fcf3cc22dd3…`, plex
`sha256:e0ab27395614…`.
- **Each chosen tag resolves to the same digest** (anonymous registry HEAD, OCI index + manifest-list
Accept): `pihole/pihole:2026.10.0` = `sha256:d34134716fc3…`; `ghcr.io/linuxserver/jackett:v0.24.2865-ls7`
= `sha256:4fcf3cc22dd3…`; `plexinc/pms-docker:1.43.4.10903-e5521bd8c` = `sha256:e0ab27395614…`.
(For contrast, `pms-docker:latest` is now `sha256:6d778d41…` and `jackett:latest` `sha256:ee6ae915…`.
The pods would have drifted on their next restart.)
- `renovate-config-validator` (ghcr.io/renovatebot/renovate:44.167.0) reports "Config validated
successfully". The only warning is the existing `docker` block migration.
- **Renovate dry run** (`RENOVATE_PLATFORM=local`, `--dry-run=full`, 44.167.0, anonymous, includePaths
limited to the six files). It extracted all three from both `catalog/` (custom regex) and
`apps/wave-0/` (kubernetes) under the same depName, so each lands in one branch:
- plex `1.43.4.10903-e5521bd8c` -> `1.43.5.11079-b3b13e506` (patch, `renovate/plexinc-pms-docker-1.x`)
- jackett `v0.24.2865-ls7` -> `v0.24.2961-ls20` (patch, `renovate/lscr.io-linuxserver-jackett-0.x`).
This needs the ghcr override; without it there was no proposal.
- pihole `2026.10.0` -> none. That is correct, because 2026.10.0 is the newest numbered tag on Docker Hub.
- **Automerge, schedule and dashboard flags** come from Renovate's own `applyPackageRules`
(datasource-merge, then update-type-merge, the same order as `flatten.js`), run on the resolved
config. The local platform does not run the branch worker:
- plex patch: automerge **true**, daily window.
- jackett patch: automerge **true**, schedule **sunday** window.
- pihole (synthetic updates) `2026.10.1` patch and `2026.11.0` minor: automerge **true**. `2027.01.0`
major: automerge **false**, `dependencyDashboardApproval` **true**, +needs-review/breaking.
- **Versioning filters** (Renovate's versioning module): the jackett regex rejects `latest`, bare
`0.24.2961`, `amd64-…` and `development-…`. The plex regex rejects `latest`, `public`, `beta`,
`plexpass` and `…-amd64`/`…-arm64`. Docker versioning ranks `2026.09.0 < 2026.10.0`.
- Touched paths (`catalog/`, `apps/`, `renovate.json5`, `runbooks/`, `decisions/`) are not in
`infra/gitea/approval-policy.json` `sensitive_paths`.
## What I did NOT verify
- **That the merge is restart-free. It is not.** The pod-template `image` string changes, so Flux
rolls all three: plex and jackett are `Recreate`, so each has a brief outage, and pihole is a
2-replica StatefulSet that rolls one pod at a time. The digest is identical, so the image's
behaviour should not change, but the kubelet resolves the new tag name and may pull it. The
rollout itself was not observed.
- The implicit `imagePullPolicy` moves from `Always` (`:latest` or no tag) to `IfNotPresent`. This is
intended, and was not observed live.
- A real (non-dry) Renovate run against Gitea, and whether the Sunday window actually opens a jackett
PR. The schedule was only evaluated, not exercised.
- Other `lscr.io/linuxserver/*` images (radarr, sonarr, qbittorrent) may hit the same 20-page
truncation once their tag counts grow. Not measured here, and not changed.
- Plex not self-updating on a version tag was checked only against the running pod (read-only
`kubectl exec cat`). `/etc/cont-init.d/50-plex-update` does `exit 0` when `version` from
`/version.txt` equals the installed `plexmediaserver` package. Both are
`1.43.4.10903-e5521bd8c`. I did not boot a pod on a *new* tag to watch that short-circuit.
+101
View File
@@ -0,0 +1,101 @@
---
date: 2026-10-30
branch: feat/pm-agent-registry-pull-secret
title: PM agent — #61 rebased onto the CI image pipeline, first published digest pinned, and the remaining path to live audited
---
## What changed
PR #61 (`feat/pm-agent-registry-pull-secret`, the kubelet pull credential and admission rule
7) was rebased onto `origin/main` after #81 (CI image pipeline), #82 and #83. Conflicts in
`docs/plans/pm-agent-issue-to-pr.md`, `platform/pm-agent/README.md` and
`runbooks/pm-agent.md` were resolved keeping both sides: #61's "pull credential now wired"
and #81's "CI publishes, Renovate bumps". No manual build-and-push text came back.
On top of the rebase, as a second commit:
- **The first published digest is pinned**: `deployment.yaml` and `PM_WORKER_IMAGE` in
`configmap.yaml` now name
`gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`
(images run 340, tag `sha-523cd9118f95`, which is also `main`). The `# renovate-digest:`
markers are unchanged and still sit on the line directly above each value, so later bumps
stay Renovate's. Pinning by hand once removes the dependency on Renovate's first-ever run.
- **The first run is dry, and scoped to a label**: `PM_DRY_RUN: "true"`,
`PM_REQUIRE_LABEL: "agent"` (the plan's follow-up 3); `config-generation` bumped to `2`.
Dry run is also a *correctness* requirement, not only caution: the pinned digest was
built before this PR added `imagePullSecrets` to the worker Job template, which is baked
into the image. Admission now requires that entry, so a worker Job built from the run-340
image would be denied. Merging this PR touches `infra/agent/**`, so `images.yaml`
re-publishes agent-runner; Renovate's bump to that digest is what makes workers
admissible. Only then flip `PM_DRY_RUN` to `false`.
- **The token scope list was wrong; it is fixed.** Gitea 1.24.3's `routers/api/v1/api.go`
closes the `/user` route group with `tokenRequiresScopes(AccessTokenScopeCategoryUser)`,
and the dispatcher's first call is `GET /user`. The documented scopes are now
`read:user`, `write:repository`, `write:issue`, `read:package` everywhere (both
`.example`s, README, runbook row).
- **Org-package permission is documented separately from the scope.** Per
`services/context/package.go` `determineAccessMode` (v1.24.3), a non-admin's access to an
org-owned package is the highest Packages-unit mode across its org teams, else read if the
org is visible to it. Home-Network is `public`, so it works today by visibility alone; the
registry `.example` now has the operator create a `pm-agent-pull` team (no repositories,
Packages = Read) so it does not depend on that.
- **`sops` is toolbox-only**: both `.example`s now say so and give the in-place
`./infra/devcontainer/dev.sh run sops <file>` edit (the same procedure
`runbooks/vpn-remote-access.md` documents), which writes no plaintext file.
- The runbook's Secret-key check used `-o jsonpath='{.data}'`, which prints values before the
pipe trims them; it is now a `go-template` that emits key names only. Two new runbook
rows: the `read:user` 403, and the admission denial from a pre-#61 digest.
## What I verified
- **Registry digest**, read-only from the registry API (bearer token from `/v2/token`,
`HEAD /v2/home-network/agent-runner/manifests/<tag>` with OCI and Docker accept headers):
`main` and `sha-523cd9118f95` both answer `200`,
`Docker-Content-Digest: sha256:c4eba5b3…6eb42`, `application/vnd.oci.image.manifest.v1+json`.
Tags list: `[main, sha-523cd9118f95]`. The admission pattern
`gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*` matches the pinned value.
- **#81's claim that the template's digest needs no tracking holds**: `pm-dispatch.build_job`
sets `containers[0].image = cfg.worker_image` (from `PM_WORKER_IMAGE`) on every Job.
Renovate's `renovate-digest` custom manager only matches `platform/pm-agent/*.yaml` and
`versions.env`, so the template is not touched by it.
- **Node path to the registry, read-only over SSH, all three prod nodes**: `/etc/hosts` line 9
is `192.168.2.20 keycloak.home.keithjackson.dev gitea.home.keithjackson.dev`, nsswitch is
`hosts: files dns`, and `curl https://gitea.home.keithjackson.dev/v2/` from the node answers
`401` with `ssl_verify_result=0` and `remote_ip=192.168.2.20`. That is the host resolver,
which is what containerd (a node process) uses; a pod's view would not have shown it.
- **Network path from the namespace**: dispatcher and worker both use
`PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000` (ConfigMap, Job template, and
`pm-worker`'s hard allowlist, which refuses anything else); the git remote and the API base
are both derived from it. So nothing goes through the Gateway and #83's
`toEndpoints`-through-Gateway issue does not apply. Egress: `toEndpoints` gitea pods `:3000`
(both CNPs), `kube-apiserver` (dispatcher only), `world:443` (the model). Ingress on the
Gitea side: `gitea-allow-pm-agent-ingress` exists live and is `VALID True`.
- **Live state (read-only, `--context k3s-prod`)**: namespace `pm-agent` exists; Deployment
`0/1`, pod `ImagePullBackOff` for 24d on the all-zero digest; Secret `pm-agent-credentials`
(2 keys) exists, `pm-agent-registry` does not yet (it ships in this PR); Flux Kustomization
`pm-agent` Ready at `main@dfc432c`; ClusterPolicy `pm-agent-job-shape` Ready.
- **Gitea state (read-only API)**: 1.24.3; org Home-Network `public`; no user `pm-agent`
exists; teams are `Owners` and `registry-push`; collaborators are `jakman85` (owner) and
`renovate-bot` (write); the repository has **no labels** and **no open issues**.
- **Placeholders, without decrypting**: the registry ciphertext decodes to 155 bytes, exactly
the length of the `.example`'s placeholder JSON. The credentials values decode to 57 and 39
bytes — neither a 40-char Gitea token nor a ~100-char Claude OAuth token. Both files'
`lastmodified` is the commit that created them as placeholders.
- `kyverno test`, `PM_SELFTEST=1 pm-dispatch` and the gate: numbers in the PR description and
the final handoff report for this run.
## What I did NOT verify
- **Any authenticated pull, poll or worker.** No `pm-agent` account or token exists, so the
scope set has not been exercised end to end. The `read:user` requirement is read from
Gitea's v1.24.3 source, not from a live 403.
- **That the credentials ciphertext is a placeholder.** Lengths are strong evidence, not
proof; I did not decrypt.
- **That Renovate will open the post-merge bump.** Its token was being re-minted at the time
of writing and it has never completed a run here. Renovate's `kubernetes` manager also
scans `infra/agent/*.yaml` and `platform/**`; what it does with a digest-only reference
with no tag (the template) is unmeasured.
- **The dispatcher's `claude --print` call to `api.anthropic.com`** through `world:443`, and
whether the run-340 image's dispatcher behaves as this tree's code does (it predates #61's
selftest line, nothing else in `pm-dispatch.py`).
- **The admission policy against a real Job.** Still never admitted or refused one live.
+108
View File
@@ -0,0 +1,108 @@
---
date: 2026-10-30
branch: fix/renovate-auth
title: Renovate's token needs five scopes, not three; the workflow documented the wrong list, and now probes it
---
## What changed
**The fix is operator-side.** `RENOVATE_TOKEN` lacks `read:organization`, and it will next lack
`write:issue`. The operator must re-mint it with five scopes:
- `read:user`
- `read:organization`
- `write:repository`
- `write:issue`
- `read:package`
The git half fixes the two things that sent the operator round this loop twice.
**`.gitea/workflows/renovate.yaml`**:
- **The header's scope list is corrected**, with the call each scope serves. The old list named
only `write:repository`, `read:user` and `read:package`, and the operator minted exactly that.
The preflight's "TO FIX" text is corrected the same way.
- **A new step, "Preflight - the token has every scope Renovate needs".**
- It probes read-only GETs: `/user` for read:user, `/orgs/Home-Network` for read:organization,
`/repos/{repo}` for read:repository, `/repos/{repo}/labels` for read:issue, and registry
`/v2/token` for read:package.
- It reads the token's own scope list out of Gitea's 403 from `/admin/orgs` (names only), and
requires `write:repository` and `write:issue` in it.
- It says what a failure means: 401 is the value; a Gitea 403 is a scope, and the probe prints
Gitea's message; an "Access denied" 403 is a Cilium/Envoy denial.
- The token is fed to curl on stdin (`--config -`), never in argv.
**`ci/lint.sh`**: a new guard rejects any non-comment `uses:` in `.gitea/workflows` that needs a
Docker daemon: `docker://`, `renovatebot/github-action` and Docker's daemon-bound
`docker/*-action`s. That is the class of bug fixed in #84.
### Root cause, with evidence
Run 350 (task 567, 06:49 UTC) **got further than 348 and 349.** Gitea's router log, filtered to
the act-runner pod `10.0.0.38`:
06:49:31 GET /api/v1/user 200 OK
06:49:31 GET /api/v1/version 200 OK
06:49:31 GET /api/v1/repos/Home-Network/home-network-k3s-impl 200 OK
06:49:31 GET /api/v1/orgs/Home-Network 403 Forbidden
There were no `/v2/` requests: Renovate stopped before extraction. From the job log, verbatim:
ERROR: Repository has unknown error (repository=Home-Network/home-network-k3s-impl)
"url": "https://gitea.home.keithjackson.dev/api/v1/orgs/Home-Network",
"message": "token does not have at least one of required scope(s), required=[read:organization], token scope=read:package,write:repository,read:user"
"x-envoy-upstream-service-time": "27", "server": "envoy"
INFO: Repository finished ... "result": "unknown-error", "exitCode": 8
**This is not Cilium.** The body is Gitea's JSON scope message, Gitea logged the request, and
`x-envoy-upstream-service-time` shows that Envoy forwarded it upstream.
**The secret WAS updated.** Runs 348 and 349 got `GET /user 403` from `tokenRequiresScopes`. Run 350
got `/user 200`, and its token scope list now includes `read:user`.
**Why `read:organization` is needed.** Renovate 44.167.0, `lib/modules/platform/gitea/index.ts`
`initRepo`, calls `helper.isOrg` → `GET /orgs/{owner}` and rethrows on error. Gitea 1.24.3,
`routers/api/v1/api.go`, puts `/orgs/{org}` behind
`tokenRequiresScopes(AccessTokenScopeCategoryOrganization)`.
**Why `write:issue` is needed next.** In the same `api.go`, `/repos/{o}/{r}/issues…`, `/labels`
and `/milestones` sit in a group with `tokenRequiresScopes(AccessTokenScopeCategoryIssue)`
(lines 1466–1582). Renovate's gitea-helper uses `/issues`, `/issues/{n}/labels`, `/labels` and
`/issues/{n}/comments`, for the dependency dashboard (`:dependencyDashboard` is extended) and for
PR labels. A dry run only reads them; a scheduled run writes them.
The other causes the coordinator listed are ruled out:
- **Schedule window and dry-run.** The run never reached branch processing.
- **Repository config.** The run never reached the clone (`"cloned": undefined`).
- **The Summary step.** It is not involved; the failure is Renovate's own exit 8.
## What I verified
- I read the whole run 350 job log (204 lines), and the router lines above. Runs 348 and 349 are
as recorded in `decisions/2026-10-30-renovate-first-run.md`'s follow-up: `/user` got 403 from
`tokenRequiresScopes`.
- **The probe step, in the pinned image.** I extracted the step body from the YAML and ran it in
`ghcr.io/renovatebot/renovate:44.167.0` (`curl` 8.5.0 and `jq` are present) with a deliberately
**bogus** token. Every probe reported 401 with the "token VALUE" diagnosis, the scope-list read
fell back to a warning, and the step exited 1.
- **The scope-list parser.** Against run 350's exact message format it reports `write:issue`
MISSING. Against a five-scope list it reports both OK. Against `all` it reports both OK.
- **The lint guard.** It flags `renovatebot/github-action` in the pre-#84 `renovate.yaml`
(dfc432c, line 152). It reports nothing on the current workflows, and it ignores commented
lines. `bash -n ci/lint.sh` passes.
- **Account facts, read-only.** `renovate-bot` is a collaborator with `write`, and the org is
public.
## What I did NOT verify
- **The probe with a real token. UNVERIFIED.** I never had the token. In particular, I have not
observed that `GET /api/v1/admin/orgs` gives renovate-bot's token a 403 with a `token scope=`
list. The source says the scope check runs before `reqSiteAdmin`. If the message format
differs, the step warns and does not fail.
- **That five scopes are enough for a full non-dry run.** They cover every helper endpoint Renovate
44.167.0 calls, mapped to Gitea 1.24.3's route categories by reading source. Branch pushes go
over git-over-HTTPS with the token; I believe that needs write:repository, but I have not
observed it.
- **Registry reads as renovate-bot.** `read:package` on a public org is expected to work. The new
probe is the first real test of it.
- **Sensitive paths.** `.gitea/**` and `ci/**` are both sensitive. CI's `approval-policy` will block
until the operator posts `/approve-override` as a PR review.
+113
View File
@@ -0,0 +1,113 @@
---
date: 2026-10-30
branch: fix/renovate-first-run
title: Renovate never started because its action needs a Docker daemon jobs here do not get; it now runs as the job container
---
## What changed
**`.gitea/workflows/renovate.yaml`**: the job no longer uses `renovatebot/github-action`. The
pinned Renovate image (`ghcr.io/renovatebot/renovate:44.167.0`, the same version the action was
pinned to) is now the job's `container:`. A `run:` step calls `renovate-entrypoint.sh renovate`
inside it. `actions/checkout` is gone, because Renovate clones the repo itself.
`GITHUB_COM_TOKEN` is wired from a new **optional** secret, `RENOVATE_GITHUB_COM_TOKEN`. The step
unsets it, and `RENOVATE_DRY_RUN`, when they evaluate to the empty string. The preflight also
prints `renovate --version`, which proves the step ran inside the pinned image.
**`renovate.json5`**: one comment now names `container.image` as the version pin, in place of
`renovate-version`. The config itself is unchanged.
### Why: the root cause
Every run of the action died before Renovate started. That includes the operator's dispatch,
run 343 (task 555, 2026-10-30 06:14 UTC), and the 11 scheduled runs before it (312–322). From
run 343's log:
RENOVATE_TOKEN is present (value not printed).
[command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** --env RENOVATE_CONFIG_FILE=/github-action/renovate.json5 ...
failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
Error: The process '/usr/bin/docker' failed with exit code 1
The action is only a wrapper around `docker run`. On this runner a job deliberately gets no
Docker daemon (`decisions/2026-10-29-image-pipeline.md`): act_runner maps `docker_host` to `-` when
`DOCKER_HOST` is `tcp://`, and the DinD daemon is privileged. So this was a
platform-compatibility failure. The run never reached authentication, token scope, config or the
network. **The re-minted token is not implicated, and nothing in this log can say whether its
scopes are right.**
A job `container:` needs no daemon. act creates it through the runner's DinD the same way it
creates the default `ubuntu-latest` image, then `exec`s each step inside it. I checked this
against gitea/act v0.261.2:
- The job container's entrypoint is `/bin/sleep` (`run_context.go`). The Renovate image has
`/usr/bin/sleep`.
- The step command files are written `0o666` (`step.go`), so the image's uid 12021 can write
`$GITHUB_STEP_SUMMARY`.
- `HOME` comes from the image env.
## What I verified
- **The root cause.** I read all of run 343's job log (162 lines, job 0; job 1 is 404). It has one
error, quoted above, and no Renovate output at all. Runs 312–322 are all `failure` from the
`schedule` event.
- **The config validates against the pinned tag.** `podman run ... ghcr.io/renovatebot/renovate:44.167.0
renovate-config-validator /tmp/renovate.json5` printed `Config validated successfully`. It also
printed the known migration diff for the top-level `docker` block.
- **A dry run with the fix extracts and looks up dependencies.** This was a local reproduction. I
made a clean snapshot of this branch's tracked files. I ran it in a podman container of the pinned
image with entrypoint `/bin/sleep`, as the image's own uid 12021 with `HOME=/home/ubuntu`.
Inside it I `exec`ed the workflow step's `run:` body, extracted verbatim from the YAML, with:
- `RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full` and `GITHUB_COM_TOKEN=` (empty, to
exercise the unset);
- a docker host rule for `gitea.home.keithjackson.dev` using the `gitea-admin` credential from
`~/.git-credentials`, passed through env and never printed.
The run exited 0 and logged `Dependency extraction complete`:
- flux 15 deps / 13 files
- github-actions 24 / 4
- kubernetes 131 / 67
- kustomize 2 / 2
- regex (both custom managers) 7 / 6
- **total 179 deps / 92 files**
It found **42 updates**: 29 kubernetes, 6 flux, 2 kustomize, 1 github-actions and 4 regex
`digest`. 7 of them are `major`.
- **The `# renovate-digest:` lookups against the Gitea registry succeed.** All three
`/v2/token?scope=repository:home-network/<img>:pull` requests succeeded, and so did
`HEAD /v2/home-network/<img>/manifests/main`. They proposed `digest` updates for
`conformance/versions.env`, `infra/agent/versions.env`, `platform/pm-agent/configmap.yaml` and
`platform/pm-agent/deployment.yaml`, all of which carry the zero placeholder digest today. The
`kubernetes` manager's copies of those images report `disabled`, which is the intended effect of
the packageRule.
- **The new pin is still maintained by Renovate.** The github-actions manager extracts
`ghcr.io/renovatebot/renovate:44.167.0` from `renovate.yaml` as depType `container`, and proposes
44.198.0.
- **The `regex` vs `custom.regex` question from #73/#77 is closed.** Both custom managers produced
dependencies.
- **Missing GitHub token.** The local run logged `GitHub token is required for some dependencies`.
That accounts for 17 deps skipped with `github-token-required`, which is why
`RENOVATE_GITHUB_COM_TOKEN` is offered as an optional secret.
## What I did NOT verify
- **The workflow on the real runner. UNVERIFIED.** My reproduction ran with podman, not
act_runner + DinD, and used `--platform=local`. The act behaviour (sleep entrypoint, `exec` as the
image user, 0666 command files) comes from reading the gitea/act v0.261.2 source, not from a run.
The first real evidence will be a `workflow_dispatch` after merge.
- **The Gitea platform path and `RENOVATE_TOKEN`'s scopes. UNVERIFIED.** I have no access token, only
the `gitea-admin` password, which Renovate's Gitea platform cannot use. So these were not
exercised: platform init (`GET /user`, which needs `read:user`), repo clone, and the registry
lookups using the **token** with `read:package` rather than a password.
- **Behaviour inside the schedule window.** A dispatch outside 03:00–06:00 America/New_York
(07:00–10:00 UTC during EDT) will extract and look up, but will log "not within schedule"
instead of `DRY-RUN: Would create ...` lines. I did not change this.
- **Renovate as uid 12021 doing artifact updates** (for example helm lockfiles) on a real,
non-dry run. `--dry-run=full` does not run the tools.
- **The self-bump.** Renovate will propose `ghcr.io/renovatebot/renovate` 44.167.0 → 44.198.0 as a
minor, which the default rule auto-merges. `.gitea/**` is a sensitive path, so it will in fact
wait for an administrator override. Nothing re-runs `renovate-config-validator` on that bump,
and no gate check runs the validator.
- **No alert for "Renovate red every day".** That gap is still open. This failure ran for 12 days
without anyone noticing.
- **Sensitive path.** `.gitea/**` is in `infra/gitea/approval-policy.json`, so CI's
`approval-policy` stage will block until the operator posts `/approve-override` as a PR review.
@@ -0,0 +1,52 @@
---
date: 2026-10-30
branch: fix/renovate-hold-auth-and-self
title: Operator decision — the oauth2-proxy sidecar and Renovate's own version never auto-merge
---
## What changed
`renovate.json5` gains two packageRules. Both set `automerge: false` and `addLabels: ["needs-review"]`.
They sit after the auto-merge rule, so they override it at every update type. Each rule's
`description` carries the operator's reason.
1. **`quay.io/oauth2-proxy/oauth2-proxy`.** This is the single depName emitted for:
- `catalog/render.py` `OAUTH2_PROXY_IMAGE` (custom regex manager);
- the five rendered `apps/wave-*` sidecars;
- `platform/crafty/deployment.yaml` (kubernetes manager).
**Reason:** since v7.13.0, oauth2-proxy validates refreshed OIDC sessions against the access
token, not the id_token. Every sidecar sets `--cookie-refresh`, and that path is untested against
this Keycloak: the platform oauth2-proxy runs v7.15.5 without `--cookie-refresh`.
`render-golden` and `--selftest` pass on the bump, but both check rendered text, not the binary.
2. **`ghcr.io/renovatebot/renovate`.** This is the job `container.image` in
`.gitea/workflows/renovate.yaml`. The rule also lists `renovatebot/github-action`, so a
reintroduction would be held too. It is not used anywhere today; `ci/lint.sh` rejects it.
**Reason:** nothing re-validates `renovate.json5` against the new version before it takes
effect. The 39.x `CONFIG_VALIDATION` failure is the precedent.
## What I verified
- **The config validates on the pinned 44.167.0.** `renovate-config-validator` printed
`Config validated successfully`. The migration diff is only the pre-existing top-level `docker`
block.
- **The flag change is exactly the intended one.** I used Renovate 44.167.0's own
`resolveConfigPresets` and `applyPackageRules` to evaluate b4ff287's config and this branch's
config against the same updates (the local dry run of the #86 config). Exactly **2 of 29**
branch rows differ:
- `ghcr.io-renovatebot-renovate-44.x`: `automerge=true labels=` → `automerge=false labels=needs-review`;
- `quay.io-oauth2-proxy-oauth2-proxy-7.x`: the same change, and every upgrade in the branch
resolves `false`, the `catalog/render.py` one included.
The other 27 rows are byte-identical.
## What I did NOT verify
- **The three GitHub-hosted branches** in run 357 (`pin-dependencies`, `kubevirt-cdi`,
`intel-intel-device-plugins-for-kubernetes-0.x`) are not in the harness table: my local run has
no github.com token. Neither rule's package names can match them.
- **A live Renovate run with this config.** The next debug-level dispatch should show
`needs-review` on both branches.
- **The oauth2-proxy refresh path itself** remains untested. Exercising it on one app is the
review the label asks for.
@@ -0,0 +1,128 @@
---
date: 2026-10-30
branch: fix/renovate-run-hygiene
title: Renovate's first completed dry runs, audited; it would have opened red PRs for every app and missed the databases. Config fixed
---
## What changed
**`renovate.json5` only.** Each change carries a comment quoting the measurement behind it.
1. **Three new regex managers, so updates edit the SOURCE.**
- `catalog/*.yaml` `source.image`.
- `catalog/render.py` `OAUTH2_PROXY_IMAGE`.
- CNPG `imageName:` under `platform/` and `apps/`.
The first two match the rendered line's depName, so the catalog edit and the generated
`apps/wave-*` edit land in **one** branch.
2. **`allowedVersions` mirroring each ServiceDefinition's `update.range`:**
- gitea/gitea `<2.0.0`
- nextcloud `<32.0.0`
- qbittorrent `<6.0.0`
- radarr `<6.0.0`
- sonarr `<5.0.0`
- semaphore `<3.0.0`
Plus CNPG postgresql `<17.0.0`, because a Postgres major is a migration, not a bump.
3. **The blast-radius never-automerge list gains** `gitea/gitea`, `gitea/act_runner` and
`ghcr.io/cloudnative-pg/postgresql`.
4. **HelmRelease range widening** (`matchManagers: flux`, ranged currentValue) is never
auto-merged and is labelled `range-widen`.
5. **Majors need a Dependency Dashboard tick** (`dependencyDashboardApproval`) before a PR opens.
6. **`flux` manager docker deps are disabled.** These are duplicates of `kubernetes` manager
extractions.
7. **`velero/velero-plugin-for-aws` uses `semver` versioning**, so its RC pin can reach a stable
release.
8. **`python` minors are not auto-merged.**
9. **Two false comments are corrected.**
- Ranges "cap Renovate at patches": false, Renovate widens them.
- `nextcloud:31-apache` and `docker:27-dind` "cannot be maintained": false, they were.
PR #85 is merged, so nothing here overlaps an open branch.
### The audit: dispatch runs 353 (07:12 UTC) and 354 (07:23 UTC), on 593d0fa
Both runs ended `success`, Renovate `"result": "done"`, `exitCode 0`. Both were at `info` level, so
I reproduced the per-dependency detail with a debug-level local dry run of the same commit (see
below).
The runs were inside the 03:00–06:00 America/New_York window, so branches were processed:
- run 353: 37 "Would commit" branches;
- run 354: 40, because the operator had added `RENOVATE_GITHUB_COM_TOKEN` 9 s before it.
Gitea's router log for the act-runner pod during both runs shows 317 2xx responses. The only
non-2xx were all expected:
- `/v2/` 401: the registry challenge;
- `/admin/orgs` 403: the #85 scope probe, by design;
- 6× `HEAD …/manifests/sha256:000…` 404: the placeholder digests.
| # | Class | Count | Cause | Severity | Fix |
|---|---|---|---|---|---|
| 1 | App-image PRs edit only generated `apps/wave-*` | 9 branches | A ServiceDefinition has no `apiVersion`, so the `kubernetes` manager extracts 0 deps from `catalog/`. The oauth2-proxy sidecar is a Python constant (`catalog/render.py:132`) | **Real problem.** Every such PR fails `ci/render-golden.sh` (reproduced: rc 1 "GOLDEN DRIFT"), stays red and holds a PR slot | git (1) |
| 2 | CNPG `imageName` untracked | 3 Clusters | The `kubernetes` manager reads `image:` only. The one tracked postgres line was the grant Job (`nextcloud-db.yaml:303`, "keep in step with spec.imageName") | **Real problem.** A real run bumps the Job and leaves the databases behind | git (1) |
| 3 | Bogus or out-of-policy majors | 5 branches: gitea 28.2.0, qbittorrent 20.04.1, radarr 6.x, nextcloud 35, postgresql 18.6 | Odd legacy tags, plus `update.range` in `catalog/` not being read by Renovate | Risk if run for real: unmergeable PRs, which hold slots | git (2) |
| 4 | Auto-merge of the merge gate's own host | gitea 1.24.3→1.27.3, act_runner 0.2.11→0.6.1, both "minor" | Not in the blast-radius list. A 0.x minor is breaking by semver | **Risk if run for real** | git (3) |
| 5 | Range widening auto-merged | external-dns, headlamp (minor); longhorn (blast radius); kube-prometheus-stack (major) | Renovate widens `>=a <b`. The config's comment claimed the opposite | **Risk if run for real:** silently lets Flux adopt new minors | git (4, 9) |
| 6 | Majors hold PR slots | 10 majors in run 354 against `prConcurrentLimit: 5` | Majors never auto-merge, so they stay open | Risk: starves auto-mergeable PRs | git (5) |
| 7 | `Cannot find replaceString … alpine:3.22` (INFO) | 1 per run | The same HelmRelease `image:` line is extracted by `flux` and `kubernetes` (also the velero plugin) | Cosmetic. The branch is still right | git (6) |
| 8 | velero-plugin-for-aws stuck on `v1.14.3-rc.2` | 1 dep, 0 updates | Docker versioning treats `-rc.2` as a suffix, so stable `v1.14.3` and `v1.14.4` never qualify | **Real problem** (silent) | git (7) |
| 9 | python 3.13→3.15 as auto-mergeable "minor" | 2 files | Language runtime minor | Risk if run for real | git (8) |
| 10 | `WARN GitHub token is required` | run 353: 5 packages, 17 deps | No github.com token | Real (they were unmaintained) | **operator, done:** run 354 has no WARN and adds `pin-dependencies`, `kubevirt-cdi`, `actions-checkout-7.x`, `major-github-artifact-actions` |
| 11 | Floating or untagged images (`invalid-value`) | 4: jackett, plex `latest`; pihole, keycloak no tag | No version to compare | Real, already documented | none in git. Pinning by hand is an operator/PM decision |
| 12 | `runs-on: ubuntu-latest` `invalid-version` | 11 | The github-actions manager extracts runner labels | Cosmetic, debug-only | none |
| 13 | `Docker Manifest is unknown` (debug) | 3 | HEAD on the all-zero placeholder digests | Expected. Clears when the own-image digest PRs merge | none |
| 14 | `Docker: error fetching data from DockerHub` 403 (debug) | 5 | Hub refuses `page=11`. Renovate already has the tags | Cosmetic. Updates were still found | none |
| 15 | `Config migration necessary` | 1 | Top-level `"docker": {"enabled": true}` | Cosmetic, pre-existing | none (as before) |
**Checked and fine:**
- The 3 own-image digest updates were found in both runs. All 4 `# renovate-digest:` markers parse.
- No marker failed to parse.
- No package file is under `conformance/runs` or `tests`, nor is `DECISIONS-LOG.md`, `decisions/` or
any `*.sops.yaml` (77 package files checked).
- `kubevirt-cdi` grouping works.
- The schedule window behaved as designed.
- Vulnerability alerts: none in either log.
## What I verified
- I read both job logs in full (132 and 129 lines), and the router log for 10.0.0.38 during both
runs.
- **`renovate-config-validator` on 44.167.0**: `Config validated successfully`. The migration diff
is only the pre-existing `docker` block.
- **Local dry runs.** Pinned image, `--platform=local`, debug, origin/main snapshot. "Before" is
593d0fa and "after" is this config; the GitHub-hosted deps were skipped in both.
- **Extraction:** `regex` went from 6 files / 7 deps to **19 files / 20 deps**. Every other
manager is unchanged: flux 15, github-actions 24, kubernetes 131, kustomize 2.
- **Branches:** 33 before, **29 after**. Removed: gitea-28.x, qbittorrent-20.x, radarr-6.x,
nextcloud-35.x and postgresql-18.x. Added: velero `v1.14.3-rc.2 → v1.14.4` (patch).
- **Gitea, qbittorrent, radarr, sonarr, semaphore and seerr branches** now each carry
`catalog/<svc>.yaml` + `apps/wave-*/<svc>.yaml`.
- **The oauth2-proxy branch** carries `catalog/render.py` plus the 5 rendered apps and
`platform/crafty`.
- **The postgresql-16.x branch** carries all three Clusters.
- The 2 flux docker duplicates report `disabled`.
- **render==golden holds for the new branch shape.**
- In the toolbox, I applied the gitea branch edit (catalog + apps) and the oauth2-proxy branch
edit (`render.py` constant + 5 apps) textually. `./ci/render-golden.sh` reported
`all 10 onboarded service golden(s) reproduce`, rc 0.
- Negative control, the old shape with only `apps/wave-0/gitea.yaml` bumped: rc 1,
`GOLDEN DRIFT`.
- Both edits were reverted afterwards.
## What I did NOT verify
- **The effect of the automerge, label and dashboard-approval rules (3, 4, 5, 8). UNVERIFIED in a
run.** `--platform=local` stops before branch processing, so I saw only which updates exist, not
their merged flags. The matchers are standard (`matchPackageNames`, `matchManagers`,
`matchCurrentValue`, `matchUpdateTypes`). The next dispatch's debug log shows them per branch.
- **`allowedVersions` under docker versioning** used Renovate's "Falling back to npm semver
syntax" path (16 debug lines). That it filters is shown by the five removed branches.
- **render==golden for the other app branches** (qbittorrent, radarr, sonarr, semaphore, seerr). I
proved it only for gitea and oauth2-proxy. They have the same shape: one `source.image` line ↔
one rendered `image:` line.
- **The `allowedVersions` rules duplicate `catalog/*/update.range`** and can drift from it. Nothing
checks them against each other. A lint comparing the two is the obvious follow-up.
- **Any real (non-dry) run.** None has happened. PR creation, labels, auto-merge through Gitea's
API and the Dependency Dashboard issue are all unobserved.
- **Sensitive path?** `renovate.json5` is not in `infra/gitea/approval-policy.json`, as far as I
know. CI's `approval-policy` result on the PR is the measurement.
@@ -0,0 +1,113 @@
---
date: 2026-10-30
branch: feat/renovate-track-keycloak-operator
title: Renovate tracks the Keycloak operator (the real Keycloak version), never auto-merged; the catalog's phantom image is pinned as documentation and disabled
---
## What changed
- **`platform/keycloak/kustomization.yaml`**: each of the three `keycloak-k8s-resources/26.0.6`
release URLs (two CRDs and `kubernetes.yml`) now has a
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker on the line
above it. That is the format the existing `custom.regex` marker manager parses (the kubevirt
pattern). A comment explains that these URLs are the Keycloak version. The `Keycloak` CR has no
`spec.image`, so the operator's `RELATED_IMAGE_KEYCLOAK` picks the server image.
- **`renovate.json5`** has two new packageRules:
- **`keycloak/keycloak-k8s-resources`**: `groupName: keycloak-operator`, `automerge: false`,
`addLabels: [needs-review, blast-radius]`. A comment explains why. The operator, the CRDs and
the server are released in lockstep. The IdP gates SSO for every app and for kubectl. And
`passkey-reconciler.yaml` encodes Keycloak 26.0.6 Admin API shapes that the static gate never
exercises. Majors still go through the existing dashboard-approval rule.
- **`quay.io/keycloak/keycloak`**: `enabled: false`. Its only occurrences are in files that
deploy nothing.
The "STILL CANNOT FIX" comment also gains a one-line correction for keycloak.
- **`catalog/keycloak.yaml`**: `source.image` changed from `quay.io/keycloak/keycloak` (no tag) to
`quay.io/keycloak/keycloak:26.0.6`. A comment says the image is DOCUMENTATION-ONLY and that the
operator chooses the real one. `apps/wave-1/keycloak.yaml` was re-rendered, which changes one
line.
- **`platform/keycloak/README.md`**:
- Corrects the false claim "Renovate maintains the version". Until now Renovate could not see
the version.
- Adds a new section, **"Upgrading Keycloak: the operator IS the version"**. It lists the 26.0.6
assumptions in the passkey reconciler to diff against a new tag, and the post-merge checks.
**The phantom: pin and disable, not delete.** `apps/wave-1/keycloak.yaml` is not listed in
`apps/wave-1/kustomization.yaml`, and render-golden excludes it (`RENDER_GOLDEN_NOT_ONBOARDED`).
Deleting it would still not be a no-op:
- It is the fixture for the keycloak cases in `apps/wave-1/tests/app-slis.test.yaml` (the
StatefulSet-based `keycloak-down` rule). Those cases are discovered by the
every-PrometheusRule-in-apps/wave-1 convention.
- Removing it means moving that rule and its tests to `platform/keycloak`. That is a separate
observability change, not a Renovate one.
Pinning plus `enabled: false` changes one line in a file nothing applies. It cannot affect the
cluster, and render-golden stays green.
## What I verified
- **Live state, prod, read-only kubectl, 2026-10-30.**
- Keycloak CR `keycloak` has an empty `spec.image`.
- The operator Deployment runs `quay.io/keycloak/keycloak-operator:26.0.6`, imageID
`sha256:8e3a1a56346f…`, with `RELATED_IMAGE_KEYCLOAK=quay.io/keycloak/keycloak:26.0.6`.
- `keycloak-0` and `keycloak-1` both run `quay.io/keycloak/keycloak:26.0.6`, imageID
`sha256:a93d22e13b86…`.
- There is no PrometheusRule containing "keycloak" in any namespace. This confirms the phantom
deploys nothing, including its `keycloak-down` rule (see the next section).
- **`renovate-config-validator`** on the pinned `ghcr.io/renovatebot/renovate:44.167.0`:
`Config validated successfully`. The only migration note is the pre-existing `docker` block.
- **Local dry run** (`RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full`, debug, the same pinned
image, this worktree, with a GitHub token passed by environment variable; the token string
occurs 0 times in the log):
- The regex manager went to 21 files / 24 deps.
- `keycloak/keycloak-k8s-resources` was extracted three times from
`platform/keycloak/kustomization.yaml`, each at `currentValue 26.0.6`, datasource
`github-tags`, versioning `semver`.
- Each extraction has one update, `26.8.1`, updateType `minor`, and all three share **one**
`branchName: renovate/keycloak-operator`.
- `quay.io/keycloak/keycloak` was extracted from `catalog/keycloak.yaml` (regex) and from
`apps/wave-1/keycloak.yaml` (kubernetes) at 26.0.6, with `skipReason: disabled` and no
updates.
- **Rule effects.** `--platform=local` does not print each branch's merged flags. So I ran
Renovate's own `applyPackageRules` (from `dist/util/package-rules` in the same 44.167.0 image)
over this `renovate.json5` for each case:
| Update | Result |
|---|---|
| operator minor 26.0.6 → 26.8.1 | `automerge:false`, `groupName:keycloak-operator`, labels `dependencies, automated-update, needs-review, blast-radius` |
| operator patch | same as the minor |
| operator major | `automerge:false`, `dependencyDashboardApproval:true`, labels add `breaking` |
| `quay.io/keycloak/keycloak`, catalog and rendered | `enabled:false` |
| control: sonarr patch | `automerge:true`, so the rule did not disable auto-merge globally |
- **render-golden, host:** `render.py --selftest` passed, and `all 10 onboarded service golden(s)
reproduce`. Before the pin, the committed phantom was byte-identical to the canonical render. I
re-rendered it after the pin.
- **Toolbox gate** (commit 24db269, worktree with the ENGINE shim mounting `.git` and the spec
repo, `DEV_KUBECONFIG=/dev/null`): **rc 3**.
- The only stage that did not run is **`gitea-settings`**.
- spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all passed.
- L2/L3 `conformance` scaffold-skipped, as expected.
- **Not green.**
- **Sensitive paths:** none touched (checked against `infra/gitea/approval-policy.json`), so no
`/approve-override` is needed.
## What I did NOT verify
- **A real Renovate run.** No PR was opened by Renovate. The labels and `automerge:false` are
proven by Renovate's own matcher, not by an opened PR.
- **That 26.8.1 is safe to adopt.** I did not diff 26.0.6 against 26.8.1. That includes the
reconciler's assumptions: `recurseExecutions` displayName, `updateExecutions`, the
`RpEntityName` guard, `requiredActions` REPLACE, and config aliases. The README section lists
them for whoever reviews the first bump PR. It also says to take or confirm a fresh CNPG backup
first, because a server minor may migrate the schema.
- **FINDING, not fixed: Keycloak has no deployed "IdP down" alert.** The `keycloak-down` SLI rule
exists only inside the undeployed `apps/wave-1/keycloak.yaml`. promtool tests it, but it is not
on the cluster: there is no keycloak PrometheusRule live. Separately, because that file exists,
`render.py --emit-waves` would treat keycloak as onboarded. It would then add a Deployment named
`keycloak` beside the operator's StatefulSet (the known `--emit-waves` hazard). The proper fix is
a separate PR: move the rule and its tests into `platform/keycloak`, then delete the phantom.
- **The catalog tag can drift.** Nothing ties `catalog/keycloak.yaml`'s documentation tag to the
operator URLs. It must be updated by hand after an operator bump, and the README says so.
- **Concurrent branches** `fix/renovate-hold-auth-and-self` and `feat/pin-pihole-jackett-plex`
also edit `renovate.json5`. Neither had merged when I checked. A rebase may be needed.
@@ -0,0 +1,41 @@
---
date: 2026-10-31
branch: chore/pm-agent-digest-99
title: Pin pm-agent dispatcher and worker to the agent-runner digest built from PR #99's merge
---
## What changed
- `platform/pm-agent/deployment.yaml`: the dispatcher `image:` moves from
`agent-runner@sha256:df36d9b7…` (images run 383) to
`agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
`config-generation` goes from "4" to "5" so the pod rolls.
- `platform/pm-agent/configmap.yaml`: `PM_WORKER_IMAGE` gets the same digest. A comment line
records where it came from. The `# renovate-digest:` marker still sits directly above both
values.
- No other values changed. `PM_DRY_RUN` stays "false". `PM_DECISION_MAX_TURNS` is not set, so
it keeps its in-code default of 1. No `*.sops.yaml` was touched.
## What I verified
- PR #99's merge commit is `466f63ec3c3df968a1dab25a8811dcd41961efde` (origin/main HEAD at
branch time).
- The Gitea Actions tasks API shows the `images` workflow (images.yaml) on that commit as
**run 394** (task 643), with event `push` and status `success`.
- I resolved the digest read-only from the registry by HEADing the manifests with OCI and
Docker Accept headers. Both `agent-runner:sha-466f63ec3c3d` and `agent-runner:main` returned
`200`, an `application/vnd.oci.image.manifest.v1+json` content type, and
`docker-content-digest: sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
The two are identical.
- Toolbox gate: the rc and skipped stages are in the PR description.
## What I did NOT verify
- That the dispatcher actually rolls to the new digest after merge, and that Flux applies it.
Flux's `lastAppliedRevision` and the pod's image were not checked. That is post-merge work.
- That the new image surfaces the triage failure's real cause. It will show only on the next
failing triage, in the dispatcher log.
- That admission (rule 7, `imagePullSecrets`) admits worker Jobs from this digest. The
template comes from `infra/agent/` at PR #99's merge, which still carries the
`imagePullSecrets` entry, but no Job was created to prove it.
- I did not run the image's own selftest against this digest.
+55
View File
@@ -0,0 +1,55 @@
---
date: 2026-10-31
branch: chore/pm-agent-go-live
title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
---
## What changed
- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
`PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
other knob are unchanged. The two comments that said dry run was the only mode that worked
(because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
PR #96 pinned.
- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
dispatcher keeps its startup config and stays in dry run.
- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
"currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
are still accurate.
## What I verified
Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
`pm-agent` and `platform-config` both at `main@1126740`:
- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
`background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
`pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
for `delete jobs.batch`.
- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
`.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
and `gitea-token`.
- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
and `pm-agent-worker-allow-egress` both report `Valid=True`.
- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
`config-generation: "3"` live. No Jobs exist in the namespace.
- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
`sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
## What I did NOT verify
- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
the committed template. Whether the API server admits the Job built from the df36d9b image's
baked template has not been tested live.
- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
through its egress policy, and push a branch and open a PR. None of this has run before.
- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
it is an operator action.
- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
`gitea-settings` not run.
@@ -0,0 +1,78 @@
---
date: 2026-10-31
branch: chore/pm-agent-runner-digest
title: Bump pm-agent to the run-383 agent-runner digest by hand; Renovate saw it but the schedule withheld it
---
## What changed
`platform/pm-agent/deployment.yaml` (the dispatcher `image:`) and `platform/pm-agent/configmap.yaml`
(`PM_WORKER_IMAGE`) move:
- from `agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`, the
run-340 publish;
- to `agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d`, the
images run 383 publish of ddd04d4.
Both `# renovate-digest:` markers are unchanged, directly above their values.
`pm-agent.homelab.keithjackson.dev/config-generation` goes `"2"` → `"3"`. That is the convention in
`deployment.yaml`'s own comment: "bump it in the same commit as a change to configmap.yaml". The
image change alone would roll the pod, but the convention is followed regardless.
No `*.sops.yaml` file is touched. Branched from 06d4a9a, which already contains the operator's #95.
### Why Renovate did not do this: the schedule, not a config defect
- **Run 385** (dispatch, 2026-10-31 16:14 UTC, on ddd04d4) was a **real** run. It logged
`Deleting orphan branch (… renovate/pin-dependencies)`, not a `DRY-RUN:` line. It ran at **12:14
America/New_York**, outside `"schedule": ["after 3am and before 6am"]`, so it created and updated
no branch.
- **The lookup DID see the new digest.** Dependency Dashboard issue #91 lists, under **"Awaiting
Schedule"**:
`chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to df36d9b`.
- **Nothing else blocked it.** There are no open Renovate PRs and no `renovate/*` branch on Gitea
(111 branches listed across all pages). The previous agent-runner PR, #90 (to c4eba5b), was
auto-merged by renovate-bot at 08:22 on 10-31. No PR limit was hit, and digests are not
dashboard-approval gated.
- **The registry, read-only, with Docker/OCI `Accept` headers:**
- `HEAD …/agent-runner/manifests/main` → `docker-content-digest: sha256:df36d9b7…771d`
- `HEAD …/agent-runner/manifests/sha-ddd04d460ef6` → the same digest.
**Without this PR** the next scheduled run (08:15 UTC, 04:15 EDT) would open it, and it would
auto-merge as a `digest` update. Ticking that "Awaiting Schedule" checkbox on #91 makes the next
run of any kind create it out of schedule.
### Found on the way: Renovate's labels have never been applied
- The repo has exactly one label, `agent`. The org has none.
- Every Renovate PR so far (#89, #90, #92, #93) was created with `"labels": []`.
- On Gitea, Renovate drops a label that does not exist in the repo. So `needs-review`,
`blast-radius`, `range-widen`, `own-image`, `dependencies`, `automated-update`, `breaking` and
`ci-tooling` are never attached.
- The **automerge flags still work.** Only the labels, the visible review signal, are missing.
- **Operator fix:** create those labels on the repo. This was not changed here.
## What I verified
- **The new image carries the fix.** I fetched both images' layers read-only from the registry and
read `/etc/pm-agent/worker-job-template.yaml` from each (`infra/agent/Containerfile:116` copies
it there):
- df36d9b's has `imagePullSecrets:` / `- name: pm-agent-registry`;
- c4eba5b's has no `imagePullSecrets` at all.
This matches `infra/agent/worker-job-template.yaml:76-77` at ddd04d4.
- **`main` = `sha-ddd04d460ef6` = df36d9b**, both measured.
- **No other file references c4eba5b.** No admission policy pins it.
- **No override is needed.** `infra/gitea/approval-policy.json` lists only `platform/kyverno/**`
under `platform/`, so `platform/pm-agent/**` is not sensitive.
- I read the whole job logs of runs 362, 367 and 385.
## What I did NOT verify
- **That the rolled dispatcher starts on df36d9b and that a worker Job is admitted.** That needs
Flux to apply this and a real dispatch. Both are operator actions.
- **That the `pm-agent-registry` Secret exists and decrypts in-cluster** (#95's sops file). I did not
read it.
- **That Renovate will treat this hand bump as current.** It should: its next lookup finds
`main` == the pinned digest, so it proposes nothing and drops the dashboard entry. Not observed.
@@ -0,0 +1,77 @@
---
date: 2026-10-31
branch: fix/pm-agent-triage-exit
title: pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible
---
## What changed
- `infra/agent/pm-dispatch.py`, `ask_pm()`: on a non-zero exit the error now comes from a new
`describe_cli_failure()`, not from `proc.stderr` alone. With `--output-format json` the CLI
reports its own failures as the result envelope on **stdout** and writes nothing to stderr,
which is why both failures today logged `claude exited 1: ` with an empty message. The new
line carries the envelope's `subtype`, `is_error`, `api_error_status`, `num_turns`,
`duration_ms`, `total_cost_usd`, `stop_reason`, its `result`/`error` text, and the tails of
stdout and stderr. It names silence explicitly when both streams are empty.
- `redact()`: everything in that message, and in `extract_decision()`'s "no decision object"
error, passes through it. It covers `sk-ant-…`, `Bearer`/`token` values, and `user:pass@` in
both `http://` and `https://` URLs.
- `PM_DECISION_MAX_TURNS`: a new knob. It defaults to 1, today's value, so behaviour is
unchanged, and it is bounded to 1–5. If the newly visible envelope says `error_max_turns`,
the remedy is a ConfigMap change rather than another image build. Documented in
`platform/pm-agent/README.md`.
- Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a
stderr-only failure, total silence, and redaction.
## What I verified
- **Timeline (Loki, `{namespace="pm-agent"}`, and `kubectl logs`):**
- **16:37:38 to 16:38:00**: pod `pm-agent-898fd8f5f-dp5hs`, image **c4eba5b3**,
config-generation 2, dry run. It triaged #97 in **22 s**, logged `PM says comment`, and
posted the 16:38:00Z comment. The thread then held only the issue body.
- **16:43:09 to 16:44:00**: pod `pm-agent-f7c9b7fcf-j8mzn`, image **df36d9b7**,
config-generation 3, **still dry run**. It logged `claude exited 1: ` with an empty
message after **51 s**.
- **16:55:42 to 16:56:36**: pod `pm-agent-7796db8bcb-p5dsm`, df36d9b7, config-generation
4, live. The same empty failure, after **54 s**.
- The only successful triage ever was the 16:38 one. Every earlier `898fd8f5f`/`5db55d558c`
log line is a startup crash (Gitea `GET /user` 401), not a triage.
- **Going live is NOT the cause.** The first failure happened in dry run. `ask_pm()` has no
dry-run branch: the prompt, flags, tools (`--tools ""`) and cwd are identical in both
modes. `PM_DRY_RUN` gates only `create_job()`, which runs after the model has answered.
- **Ruled out, with evidence:**
- **Timeout**: 51 to 54 s against a 900 s limit, and a timeout raises `TimeoutExpired`,
not rc 1.
- **Budget cap**: the PM call passes no `--max-budget-usd`.
- **Prompt size**: the thread is the issue body (293 chars) plus three comments
(1871 + 404 + 77 chars), and the persona is 8 KB.
- **Writable HOME**: HOME=`/workspace` (emptyDir), and `CLAUDE_CONFIG_DIR=/workspace/.claude`
was written at 16:56.
- **Egress**: a TLS 1.3 handshake to `api.anthropic.com:443` from the dispatcher pod
succeeds in 0.06 s.
- **Model name and CLI version**: the same `claude-opus-5` and CLI 2.1.285 succeeded at
16:38. Between c4eba5b and df36d9b, `infra/agent/` differs only by a template selftest
assertion and the toolbox base digest.
- **A hint, not proof:** `/workspace/.claude/.claude.json` in the live pod has
`cachedExtraUsageDisabledReason: 'out_of_credits'`. No token was read or printed.
- Offline selftest: `PM_SELFTEST=1` against the worktree's template and persona prints
`selftest OK`, including the 4 new assertions.
- Toolbox gate: see the PR description for the rc and the stages that were skipped.
## What I did NOT verify
- **The root cause. UNVERIFIED.** The CLI's own message was discarded, `--no-session-persistence`
leaves no transcript, and I did not run `claude` in the pod. The candidates still open:
- **Usage or rate limit on the subscription seat.** The seat is shared with the operator's
interactive use. The `out_of_credits` extra-usage flag fits this. So does a ~50 s run,
which matches the CLI exhausting its 429/529 retries.
- **`error_max_turns`.** With `--max-turns 1` and `--json-schema`, an answer that fails the
schema cannot be retried. The failing thread now invites `implement` (a branch plus a
long brief), where the 16:38 one produced a short `comment`. ~50 s is about two
generations of the 22 s success.
- **The 17:11 retry will NOT reveal the cause.** It runs the old image, which still discards
stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner,
and a follow-up PR bumps the dispatcher digest (`deployment.yaml` and the `PM_WORKER_IMAGE`
in `configmap.yaml`, plus a `config-generation` bump).
- That the CLI's error envelope carries exactly these key names. The parser takes whichever
keys are present and falls back to raw tails.
@@ -0,0 +1,54 @@
---
date: 2026-11-07
branch: feat/devcontainer-velero-cli
title: Bake the velero CLI into the dev toolbox, pinned to the prod server version
---
## What changed
- `infra/devcontainer/versions.env`: new `VELERO_VERSION=1.18.2`, with a comment explaining that
the CLI and server are version-coupled, so this pin follows the server and is not bumped on
its own. It has **no** `# renovate:` marker on purpose. When a `velero` chart bump in
`platform/velero/helmrelease.yaml` changes the appVersion, bump this pin in the same PR.
- `infra/devcontainer/Containerfile`: `ARG VELERO_VERSION`; the binary comes from the
vmware-tanzu GitHub release tarball and is installed in the same RUN as flux/helm. There is
no checksum check, which matches the other tools in that file (none of them verify one).
`velero=` is recorded in `/etc/dev-toolchain-versions`. The verify stage now checks that
`velero` is on PATH, and `check velero v${VELERO_VERSION}` asserts the output of
`velero version --client-only`.
- `infra/devcontainer/build-image.sh`: `VELERO_VERSION` added to `BUILD_ARGS`. CI reads that
list generically through `--print-build-args` (`.gitea/workflows/scripts/images.sh`), so
**no sensitive path (`.gitea/**`, `ci/**`) was touched**.
- `infra/devcontainer/README.md`: added a Backups row to the tool table.
- `runbooks/backup-failure.md`: under "First commands", a read-only
`velero backup describe --details` / `backup logs` example run through dev.sh.
## What I verified
- Live server version, read-only: `kubectl --context k3s-prod -n velero get deploy velero`
shows image `docker.io/velero/velero:v1.18.2`, and HelmRelease `velero` has
lastAttemptedRevision `12.2.3`.
- Local build: `IMAGE=localhost/home-k3s-dev:velero-wt ./build-image.sh` returned rc 0. The
verify stage printed `ok velero` (on PATH), `ok velero v1.18.2` (pin check) and
`TOOLBOX VERIFIED`. The image was built under a separate local tag and **not pushed**.
- Smoke test with that image against prod (read-only): `velero version` reports Client v1.18.2
and Server v1.18.2. `velero backup get` lists the daily backups, all `Completed` with 0 errors.
- Recurring warning (reported only, not fixed): every daily backup since
`velero-daily-20261031030050` has 1 warning. Backups up to 20261030 have 0. The warning text is:
`gitea-actions: resource: /pods name: /act-runner-5944dfdfb8-69bvq message: /Skip pod volume
data error: /pod is not in the expected status, ... phase=Failed: pod is not running`.
It is the same pod in 20261031, 20261104 and 20261108. `gitea-actions` holds 11 `Failed`
pods with reason `Evicted` from ReplicaSet `act-runner-5944dfdfb8`, ten of them started
2026-10-30T08:26Z. The running runner is `act-runner-fc597f6b5-vsvkl`. So the warning comes from a
leftover evicted (`Failed`) pod that Velero still tries to back up volumes for. Cleaning up the evicted pods is an
operator action.
- Gate: see the PR description for the toolbox `ci/gate.sh` rc and the skipped stages.
## What I did NOT verify
- The CI image build (`.gitea/workflows/images.yaml`) on this branch. It runs on push/PR and I
have not seen its result while writing this.
- An arm64 build. Only amd64 was built (`TARGETARCH` defaults to amd64).
- Any mutating velero command (`backup create`, `restore`, `delete`). None were run.
- Why only `-69bvq` of the 11 evicted act-runner pods shows up in the warning. A likely guess is
that only that pod's volumes match the backup's pod-volume selection, but I did not check.
+62
View File
@@ -0,0 +1,62 @@
---
date: 2026-11-07
branch: feat/gitea-28
title: "Gitea 1.27.3 -> 28.3.0 (upstream dropped the 1. prefix)"
---
## What changed
- `catalog/gitea.yaml`: `source.image` `gitea/gitea:1.27.3` -> `gitea/gitea:28.3.0` (still the
newest stable tag on Docker Hub and GitHub on 2026-11-07: 28.0.0, 28.1.0, 28.2.0, 28.3.0).
`update.range` `">=1.22.0 <2.0.0"` -> `">=28.0.0 <29.0.0"`.
- 28.x breaking changes handled in the env block:
- `GITEA__server__DOMAIN` **removed**: 28 no longer reads `[server] DOMAIN` (go-gitea/gitea#39400);
`ROOT_URL` was already set and is now the only source of the domain. Nothing else in the repo
reads the variable (the bootstrap script sets ROOT_URL/SSH_PORT only).
- `GITEA__actions__RUN_RETENTION_DAYS=400` **declared** (go-gitea/gitea#38855 added the 400-day
default; the spec repo states no CI-history retention requirement).
- `GITEA__security__EGRESS_MODE=lax` **declared, with no allow list.** Read from the 28.3.0
source: the Keycloak OIDC login source does NOT go through the new egress policy (goth's
openidConnect provider, default HTTP client). The policy covers webhooks, the OAuth2 avatar
fetch (`UPDATE_AVATAR` is off) and OpenID 2.0. A hostname entry would not even help for a
private address — `WithLocalNeedsIPAllow` requires an IP/CIDR entry. So the brief's
"Keycloak SSO breaks" premise does not hold for 28.3.0, and an allow-list entry would only
widen webhook SSRF reach.
- `GITEA__service__DISABLE_REGISTRATION=true` **declared** (28's new default; 1.27 defaulted it
open). OAuth2 auto-registration is gated separately and keeps working.
- `update.autoRollbackOnFailedHealth`: kept `true`, commented as **inert**. It renders only an
annotation on a Flux ImagePolicy, and `apps/` is rendered `--no-image-automation`, so no
ImagePolicy exists. The Deployment is `Recreate` with no probes, so nothing reverts the image
automatically. Rollback = restore the PVC, never an image revert onto the migrated DB.
- `apps/wave-0/gitea.yaml` re-rendered with the canonical command
(`render.py --all catalog -o apps --wave-layout --no-image-automation`, in the toolbox). Only
gitea changed.
- `renovate.json5`: gitea `allowedVersions` `<2.0.0` -> `<29.0.0`; the description no longer calls
28.x a stale tag.
## What I verified
- Docker Hub tag list and GitHub tags: 28.3.0 is the newest 28.x stable tag.
- Read the v28.3.0 source: `modules/egress/policy/policy.go`, `modules/egress/policies.go`, every
`egress.New*` caller, `services/auth/source/oauth2/providers_openid.go`,
`modules/setting/{security,actions,service}.go`, `routers/web/auth/{oauth,linkaccount}.go`,
`cmd/admin_user_list.go`, `cmd/admin_auth*.go`, the Dockerfile, and `docker/root` (the s6 setup
still runs `environment-to-ini`). The postStart hook's CLI is unchanged in 28.3.0: `admin user
list --admin` still has Username in column 2, `admin auth list` still has ID/Name in columns 1/2,
`add-oauth`/`update-oauth` keep the same flags, and curl and su-exec are still in the image.
- Live Gitea (API, 2026-11-07): version 1.27.3. Zero repo, org and system webhooks, no mirrors.
- `renovate-config-validator` at 44.167.0 (the tag renovate.yaml pins): "Config validated
successfully".
- Ran the toolbox gate on the committed tree (numbers are in the PR).
## What I did NOT verify
- **The upgrade itself.** Nothing booted 28.3.0 against a copy of the production DB. Migration
time, migration success and the post-upgrade SSO login are all UNVERIFIED, and the PR's
post-merge checklist covers them.
- **The age of the latest gitea-data Longhorn/off-cluster backup.** The cluster read failed from
the worktree (no k3s-prod context with the kubeconfig available), so the operator must confirm
it before merging.
- How the Envoy Gateway handles the new `/-/ws` WebSocket. If upgrades are not forwarded, live
notifications fall back to polling (not breaking). UNVERIFIED.
- Whether the L2/L3 conformance suite passes. It does not run in CI.
@@ -0,0 +1,51 @@
---
date: 2026-11-07
branch: feat/l0-externaldns-annotation-prefix
title: L0 guard — external-dns annotationPrefix must match every external-dns annotation in the tree
---
## What changed
`conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` gains assertion
(8). The check is already whole-repo-scoped, so it runs on every gate.
- **(8a)** `platform/externaldns/helmrelease.yaml` must set `values.annotationPrefix` explicitly,
and it must equal the check's `ANNOTATION_PREFIX` (`external-dns.alpha.kubernetes.io/`). The
check does not trust the upstream default, because that default is what changed.
- **(8b)** `extraArgs` must not carry `--annotation-prefix` or `--enable-legacy-annotation-prefix`,
so the prefix has one declared place.
- **(8c)** Every external-dns-looking annotation key under apps/, platform/ and runbooks/ must use
that prefix. The match is any `external-dns(.x)*.kubernetes.io/` key.
- **(8d)** catalog/render.py must emit no external-dns key under any other prefix.
`CONTROLLER_KEY` and `HOSTNAME_KEY` are now derived from `ANNOTATION_PREFIX`. The Service
inventory matches any external-dns prefix variant, not only the alpha one.
Why: on 2026-11-07, external-dns v0.23.0 (whose default prefix is GA) ignored every alpha
annotation. Assertions (1) to (3) stayed green, because they prove the annotations are present,
not that the controller reads them. See `decisions/2026-11-07-public-dns-wan-a-record.md`.
This branch is stacked on `fix/public-dns-wan-a-record` (PR #140). Without that fix, (8a) fails,
which is correct. Merge #140 first.
## What I verified
Run on the host (PyYAML):
- the check passes on this tree: `check OK: 62 assertions passed`.
- it **fails (8a)** with origin/main's HelmRelease, which has no `annotationPrefix`.
- it **fails (8c)** when `platform/gateway/gateway.yaml`'s `target` key is changed to the GA
prefix.
Toolbox gate: see the PR description.
## What I did NOT verify
- Whether a future chart could read the prefix from somewhere other than `values.annotationPrefix`,
for example a renamed chart value. (8a) would then pass while the flag silently vanished from the
rendered args. Rendering the chart in the gate would close that gap. Not done: it needs network
access or a vendored chart in L0.
- `conformance/**` is a **sensitive path**. This PR needs the operator's approving review or
`/approve-override`. The agent did not post one.
- The header of the sibling chainsaw-test.yaml still says "external-dns v0.21.0". That is cosmetic
and I left it alone.
@@ -0,0 +1,333 @@
---
date: 2026-11-07
branch: feat/pm-agent-pr-mentions
title: The PM agent can be @-mentioned on a pull request — an EXPLICIT mention is the only trigger, reply only, shipped OFF
---
## What changed
Issue #116 asked for the PM agent to react when it is `@`-mentioned on a pull request: read
the PR, the changes and the thread, and reply in that thread. This branch implements the
**trigger, context assembly, dispatch and reply** half. The change-authoring half (the
operator's point 6 — revise an agent-owned branch versus open a fresh branch/PR against
`main`) is deliberately absent and wants its own issue, as the PM said on the thread.
### Revised 2026-11-07: continuation removed, an explicit mention is the only trigger
The first version of this branch carried a **continuation** rule: once a thread held a
mention, every later non-bot comment in that thread triggered a run with no second mention.
The operator withdrew that requirement after reading the first PR description:
> Since there is no threads and the PR comments are flat, then the pm-agent should be looking
> for explicit mentions rather than a single mention and continued scrapping of comments
> within the PR in perpetuity
Read with their earlier *"the 'same thread' requirement was based on my assumption that
comments can be hierarchical. If not, then disregard"*, the requirement is **withdrawn, not
relocated**. So the arming state is gone, in both code and fixtures:
- `plan_pr_replies` no longer carries an `engaged`/armed flag. A comment is answered iff
(1) it is not the bot's own, (2) no earlier reply of the bot's marks it answered, and
(3) it explicitly mentions the bot account. Nothing a comment's neighbours say can make it
actionable, so a comment with no mention produces **no reply and no model call**.
- Applied **uniformly** — conversation comments, review bodies and inline review comments.
Inline comments *are* genuinely threaded in Gitea, and keeping continuation for them alone
was considered and rejected: it would be a second rule the operator did not ask for, and
the brief said to stop and ask rather than invent a middle rule. Nothing turned out to be
load-bearing on the arming state for the inline path (the inline path keys replies off the
(path, position) group, which is a *destination*, not a trigger), so there was nothing to
stop and ask about.
- Thread identity survives, but **only as a reply destination**. "What counts as the same
thread" needed a policy while it was part of the trigger; now it only answers "where does
this reply go", and the two answers are Gitea's own: the conversation, or the (path,
position) group.
- `reason` on a plan is now always `"mention"`. The field is kept because it is what the log
line and `PM_PLAN` print as *why you are running*, and the selftest asserts across every
fixture that no plan ever carries another value — a second value would mean a second
trigger had come back.
- **What a mention still batches:** two *unanswered* mentions in one thread are one reply
naming both keys, because the at-most-once-reply model is per reply, not per mention.
Comments sitting between them are neither answered nor keyed.
- The PM persona now tells the PM to **answer as if its reply is the last word it will see**,
and to ask the human to tag it again if it needs something back — under the old rule it
could end a reply with a question and expect the answer to reach it. It will not.
Kept untouched, as the brief required: the in-reply idempotency marker and the
at-least-once-decision / at-most-once-reply failure model; the self-authored filter as the
first check; the hostile-payload fencing and the untrusted banner; `PM_PR_MENTIONS: "false"`,
`PM_PLAN`, `PM_DRY_RUN` implying plan mode and the two-step rollout; the `/notifications`
query with its fallback and the `read:notification` scope note; `PM_PR_DIFF_MAX_CHARS` and
its default. **Context assembly is unchanged** — "explicit mention required" governs what
triggers a run and what gets replied to, not how much the model sees, so the payload still
carries the PR, the changed files, the diff and the whole comment list. The only payload
edit is a line telling the PM that comments which do not mention it are context, not
questions put to it.
**The dispatcher was not structured the way the brief guessed, and that is worth recording.**
The brief expected the poll loop under `ci/**` and/or `.gitea/**`. It is not there: it is
`infra/agent/pm-dispatch.py`, a long-running Python program in the pm-agent Deployment
(`platform/pm-agent/`). `ci/**` and `.gitea/**` are the merge gate and have nothing to do
with it. **So this PR touches no approval-policy sensitive path** — I checked every changed
path against `infra/gitea/approval-policy.json` by hand (see "What I verified") — and the
`ci/approval-policy.sh` stage should pass on a green gate without a human review. The PM's
own warning on the issue that the PR would sit red was an honest guess from an unverified
belief about where the code lived; it was wrong, and nothing about the change needed to move
to make it so.
There *was* a poll loop to extend, so no architecture was improvised.
`infra/agent/pm-dispatch.py`:
- A second, independently-guarded pass per poll — `cycle_pr_mentions()`, called from `run()`
rather than from `cycle()`. Two reasons: the issue cycle returns early at the worker cap
and a mention creates no worker, so it must not be suppressed by a cap it cannot
contribute to; and a Gitea failure on one path must not silence the other.
- **Discovery** (`discover_pr_candidates`). Tries Gitea's notification list for the bot
account — one bounded call, which is the "way to check for '@'s for the bot account
instead of scouring the entire pull request list" the operator asked for — and falls back
to listing open pull requests. Mode is a knob (`auto` / `notifications` / `pulls`), and
which query answered is logged every cycle. **A 401/403 is handled as its own case**
(`GiteaForbidden`) because on this service it is almost certainly a token scope, not a bug:
`/api/v1/notifications` sits behind Gitea's `notification` scope category and the bot's
token is minted `read:user, write:repository, write:issue, read:package`. The log names
`read:notification` as the remedy instead of reporting "no mentions found".
- **Detection** (`normalize_pr_entries`, `plan_pr_replies`) — pure functions over normalized
comments, so the parts that must be provable offline are provable offline. Reads all three
Gitea resources: conversation comments (`/issues/{n}/comments`), review bodies
(`/pulls/{n}/reviews`) and inline review comments (`.../reviews/{id}/comments`).
- **Context assembly** (`render_pr_thread`) — PR title and body, the changed-file summary,
a bounded diff, the full thread being answered with the anchor comment marked, and the
surrounding comments. Every field that came from a person goes through a single new
`fence()` helper under an explicit untrusted banner; `render_thread` was refactored onto
the same helper, which removes a twice-retyped `.replace("```", "`` `")`.
- **A narrowed decision shape.** `PR_ACTIONS = ("comment", "decline")` and
`PR_DECISION_SCHEMA` with **no `branch` and no `brief` property at all** — the PM has no
field in which to ask for work. `handle_pull()` refuses anything outside that set (and
still marks it answered, so an unusable decision costs one decision rather than repeating
forever), and nothing on this path calls `build_job()`.
- **Replying in the thread.** Conversation threads get an issue comment. Inline threads get
a one-comment review (`POST /pulls/{n}/reviews`, `event: COMMENT` — never `APPROVE`),
which is the only way Gitea creates an inline comment; if Gitea refuses it the reply falls
back to the conversation naming the file and position, because a mention answered in the
wrong place is better than one answered nowhere.
- **`PM_PLAN`** — discover, assemble, print the exact payload and the exact thread it would
post into; ask no model, post nothing. Setting it also enables the path, so it is one knob
to look before leaping. `PM_DRY_RUN` implies it (see "the decision I made" below).
`infra/agent/testdata/pr-mention-fixtures.json` (new) — 18 fixture cases driving the trigger
logic from `PM_SELFTEST=1 pm-dispatch`, covering the six the brief required plus twelve edges.
The three cases that asserted continuation are **inverted rather than deleted**, so the old
behaviour cannot creep back unnoticed, and three were added for the revised rule.
`infra/agent/Containerfile` COPYs them into the **verify stage only** (test data has no
business in the shipped image) and the selftest **fails rather than skips** if it cannot find
them. `infra/agent/pm-persona.md` gains a "When you are reading a PULL REQUEST" section, and
the selftest asserts it is there — without it the PM would be answering a mention while still
believing it may pick `implement`.
`platform/pm-agent/configmap.yaml` adds the knobs with `PM_PR_MENTIONS: "false"`;
`deployment.yaml` bumps `config-generation` 5 → 6 in the same commit, as that file requires.
`platform/pm-agent/README.md` and `runbooks/pm-agent.md` document the path, the token scope,
the plan mode and six new log lines.
### The decisions made explicitly
**The trigger.** Settled by the operator rather than by me: an explicit mention, and nothing
else. See the revision section above. The earlier back-and-forth about what "the same thread"
means for a flat comment list is moot — the question only existed because continuation needed
a notion of sameness, and there is no continuation.
**Idempotency ordering.** The marker lives *inside the reply the PM posts*
(`<!-- pm-agent:replied-to=comment-9001,review-comment-77 -->`), which makes it one atomic
write with the reply and therefore impossible to get out of step with it. The failure mode
chosen, deliberately, is **at-least-once model call, at-most-once reply**: a crash before
posting costs one repeated decision; a crash after posting is a no-op because the marker is
already in the thread; and a POST that errors while Gitea actually created the comment is
also a no-op, because nothing is retried inside the cycle and the next cycle re-reads the
list. A mention is therefore never silently dropped and a reply is never duplicated. The
rejected alternative — write a marker *before* asking the model — buys cheapness by making a
crash swallow the mention with nothing in the thread saying so. `PM_PR_COOLDOWN_SECONDS`
(900s) bounds the cost of the repeat, which is why paying it is the cheap side of the trade.
**Loop prevention.** With no author gate (operator answer 3: *"anyone can talk and have the
agent work"*), any passer-by can start a run, so a bug that let the bot's replies count as
triggers is an unbounded loop anyone could light. The self-authored check is the first thing
`plan_pr_replies` does with each comment and it is structural — a bot-authored comment can
only ever *clear* work, never create it, so a reply of ours containing our own handle is
inert. The posted reply additionally never contains `@`-the-bot. Fixture (c) is that case and
the selftest asserts the second guard too. Dropping continuation narrows this surface
further: a reply is now reachable only from a comment that spells the handle out, and the
bot's own output never writes it, so the guard is doubled rather than relied upon.
**Why it ships off.** Not one call on this path has been made against the live Gitea, and
the endpoint it most wants is expected to be refused on today's token. `CLAUDE.md` §2b is
explicit that this repository is not evidence about anything live, and the service's own
rollout precedent is a staged one (`PM_DRY_RUN=true` for a day, `PM_REQUIRE_LABEL=agent`).
Shipping it on would be asserting it works. `PM_DRY_RUN` was also made to imply `PM_PLAN`,
because the runbook sells `PM_DRY_RUN` as "stop it taking new work" and a path that kept
posting replies under it would be a footgun — and on this path the reply *is* the action
under test, and it also carries the idempotency marker, so posting it would make a dry run
not dry.
No requirement ID is claimed. The brief is right that nothing in the `R-*` set covers agent
tooling, and inventing one would be worse than the gap.
## What I verified
**The offline selftest, run in this container, exit 0.** 95 assertions, all passing:
PM_SELFTEST=1 PM_WORKER_JOB_TEMPLATE=infra/agent/worker-job-template.yaml \
PM_PERSONA_FILE=infra/agent/pm-persona.md python3 infra/agent/pm-dispatch.py
-> selftest OK, rc=0
`PM_PERSONA_FILE` has to be named because this worker container's own image is older than
this branch, so the baked `/etc/pm-agent/pm-persona.md` is the pre-#116 copy and the
"persona states" assertion fails against it. In the shipped image the Containerfile COPYs
the branch's persona to that path and the override is unnecessary.
All 18 fixture cases pass, including the six the brief named: (a) a mention in a PR
conversation comment is detected; (b) a mention in an inline review comment is detected as
its own thread; (c) a bot-authored comment is ignored *including when it contains the bot's
own handle*; (d) an already-marked comment id is skipped; (e) a conversation follow-up with
no mention is **not** actionable even after the bot replied; (f) an unrelated comment on an
unrelated PR triggers nothing.
The cases that pin the revised rule:
- **(e), (m), (l) — inverted, not deleted.** (e) a conversation follow-up after a reply, (m)
an inline follow-up in a thread the bot has already replied in, and (l) an unmentioned
inline comment on a pull request whose conversation was answered: all three now expect
**no reply at all**. (m) is the deliberate one — an inline thread is genuinely threaded in
Gitea and still gets no continuation.
- **(i) — the burst shape the operator described.** Three comments, only the first mentioning
the bot: exactly **one** reply, keyed to the first comment only. The other two are carried
into the payload as context and are not answered.
- **(q) — the consequence that is now closed.** A once-mentioned-and-answered pull request
then gets comments from `renovate` and `gitea-actions`: no reply.
- **(p)** two *unanswered* mentions in one thread are one reply naming both keys; **(r)** a
fresh mention after an answered one gets its own reply, keyed only to itself, so a mention
is repeatable; **(o)** a comment made before the mention is still not pulled in.
- Three assertions over **every** fixture at once, so a single case drifting cannot hide the
regression: every plan's `reason` is `mention`, every comment answered matches the mention
pattern, and no bot-authored comment is ever answered.
Also still passing, unchanged: `@pm-agent-worker` and `[email protected]` (and
`@pm-agentx`, `mail:foo@pm-agent`, `@pm-agent.worker`) are **not** mentions — which matters
more now that the mention is the only trigger; a marker pasted by a non-bot author does not
suppress a mention; a mention in a review *body* is a conversation trigger; two inline
threads on one file stay separate with the newest-anchored one taken first; a position-less
inline comment still lands in a stable thread.
Also asserted: the reply carries a marker for every key it answers and contains no
`@pm-agent`; the refusal comment is marked too; `PR_ACTIONS` is exactly `comment`/`decline`
and a subset of `ACTIONS`; `PR_DECISION_SCHEMA` has no `branch`/`brief` and
`additionalProperties: false`; `fence()` breaks an embedded triple backtick rather than
dropping it; the assembled payload on a hostile PR (a body *and* a diff that each try to
close the fence and issue orders) has balanced fences, carries the banner twice, and quotes
the hostile text verbatim inside a fence; truncation keeps both the head and the tail.
**The dispatcher plumbing the selftest does not reach, re-exercised against a stub Gitea
after the revision** (a throwaway script under `$TMPDIR`, not committed; 26 assertions, all
passing). Measured:
- discovery resolves a notification subject URL to index 120 and reports source
`notifications`; a `GiteaForbidden` from `/notifications` falls back to `open-pulls`; in
`notifications`-only mode it returns `([], 'notifications-unavailable')` rather than
silently widening into the scan the operator rejected;
- plan mode printed the payload and posted **nothing** for four fixtures;
- case (a) posted into the **conversation** carrying marker `comment-9001`, and case (b)
posted **inline** on `infra/agent/pm-dispatch.py` position 42 carrying marker
`review-comment-7701` — so the reply still goes back where the mention was made;
- case (i) (burst, only the first mentions) produced exactly one reply whose marker names
**only** `comment-9070`, and its payload still carried comments 9071 and 9072 as context;
- cases (e), (m), (l) and (q) produced **no reply and no model call** — the stub counted zero
calls into `ask_pm_pr` across all four, which is the "no reply and no model call" half of
the brief's done-criterion measured rather than asserted;
- case (p) produced one reply whose marker names both `comment-9100` and `comment-9102`;
- an out-of-vocabulary action (`implement`) was refused with a marked explanatory comment and
no Job;
- the rendered payload says `why you are running: **mention**` and carries the line telling
the PM that unmentioned comments are context, not questions put to it.
**The merge gate, in this container, re-run after the revision:**
./ci/gate.sh -> exit 3 (UNDETERMINED)
Stages that did not run: spec-immutability gitea-settings
diff base: origin/main @ 5b484f6b7884 ... HEAD @ 2fdb95393cc0
Nothing failed. Those are the two skips `CLAUDE.md` §3 names as expected here — the sibling
spec repo is not reachable from the cluster and `SETTINGS_API_TOKEN` exists only in CI — and
they are the *only* two. `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and
`approval-policy` all reported green.
**That this PR touches no sensitive path**, checked path by path against the
`sensitive_paths` globs in `infra/gitea/approval-policy.json`: the changed files are under
`infra/agent/`, `platform/pm-agent/`, `runbooks/` and `decisions/`. `infra/agent/**` is not
in the list (`infra/gitea/**` is, and is a different directory), `platform/pm-agent/**` is
not (`platform/kyverno/**` is), no `rbac.yaml` and no `*.sops.yaml` is touched, and
`platform-config/kustomization.yaml` and `CLAUDE.md` are untouched. The gate's own
`approval-policy` stage is the authority on this and it only evaluates on a
`pull_request` event, so **read its verdict on the PR** rather than trusting this paragraph.
## What I did NOT verify
- **Nothing here has run against the live Gitea, and nothing here is deployed.** I held no
Gitea credential, no kubeconfig and no ServiceAccount token: this container has egress to
the in-cluster Gitea and the Anthropic API only, and `kubectl` fails. Every claim below is
therefore an operator measurement, not mine.
- **Every one of the five new Gitea API calls is UNEXERCISED.** `/notifications`,
`/pulls/{n}`, `/pulls/{n}/files`, `/pulls/{n}.diff`, `/pulls/{n}/reviews`,
`.../reviews/{id}/comments` and `POST /pulls/{n}/reviews` were written to Gitea's published
swagger for the 1.24 line. The fixtures prove this program *parses* those shapes; they
prove nothing about whether this Gitea *emits* them. The exact read-only curl commands to
settle it are in `platform/pm-agent/README.md`, and replacing the fixtures with a real
capture is the cheapest way to close the gap.
- **That `/api/v1/notifications` needs `read:notification` is read from Gitea's documented
scope model, not measured.** It is the reason the fallback and the named-remedy log line
exist; if the endpoint turns out to work on today's token, the fallback simply never fires.
- **Whether a notification is actually raised for a mention in an *inline review* comment,
and whether `subject.url` is populated for it.** If it is not, discovery via notifications
will miss inline mentions and only the `pulls` fallback will find them. Untested; it is the
single most likely gap in the cheap query.
- **Whether `POST /pulls/{n}/reviews` with one comment lands in the *existing* inline
thread** or opens a second conversation at the same line. Gitea groups by (path, position)
in its UI, which is why this should work, but I have not seen it. The fallback covers
outright refusal, not mis-placement.
- **Whether `position` from a stored review comment is still a valid `new_position` on the
current head.** It will not be after a force-push, which is exactly when the inline
fallback fires.
- **The image.** Nothing was built. The Containerfile `verify` stage change (the fixture
COPY) has not been run by a container engine — there is none in this container — so the
only evidence the selftest passes under the fixture path resolution is the repo-checkout
run above, which takes branch 2 of `pr_fixture_path()`. CI's `images` workflow is what
exercises branch 3. Related and worth knowing when reading a failure: the selftest's
"persona states" assertion fails in *this* container against the baked
`/etc/pm-agent/pm-persona.md`, because the image predates this branch. That is an artefact
of where the run happened, not a defect — in the built image the two are the same file —
but it does mean the image's own selftest invocation (`PM_SELFTEST=1 pm-dispatch`, with no
`PM_PERSONA_FILE`) has not been observed passing with this branch's persona.
- **The two open items from the first version of this branch are closed, and that part is
verified offline rather than unverified.** They were "continuation has no end condition"
and "another bot commenting on a once-mentioned PR would get a reply". Both are gone by
construction: there is no continuation left to terminate, and fixture (q) measures that
`renovate` and `gitea-actions` commenting on an answered pull request draw nothing. No
ignore-list for automated authors was needed or added.
**The residual case, stated precisely.** A reply is still reachable by *anyone* who writes
the literal handle, including an automation, because the operator's answer (3) is "anyone
can talk and have the agent work" and there is no author gate. Two concrete shapes:
(1) a bot whose comment template happens to contain `@pm-agent` — e.g. somebody quoting a
mention into a Renovate PR description that Renovate then echoes — gets one reply per such
comment; (2) **a quoted mention counts as a mention.** Gitea gives this program no way to
tell `> @pm-agent what do you think?` pasted into a Markdown quote from a fresh request, so
re-quoting an earlier mention in a new comment costs one more decision and one more reply.
Both are bounded — one reply per comment id, one decision per pull request per 15 minutes —
but neither is prevented. Whether to ignore quoted lines, or particular authors, is a
policy call for the operator and not one I should take.
- **Spend.** `PM_PR_DIFF_MAX_CHARS` defaults to 20000, so a mention costs a decision over a
payload meaningfully larger than an issue triage's. No measurement of the real cost exists.
Dropping continuation should reduce the *number* of runs substantially — the trigger rate
is now how often somebody types the handle rather than how busy a once-mentioned pull
request is — but that is reasoning, not a measurement.
- **R-TEST-06 (spec immutability) and L2/L3 conformance** were not verified by this run —
the two gate skips above. And the `approval-policy` stage reported N/A because there was no
`pull_request` event; its verdict on the PR is the authoritative one.
@@ -0,0 +1,99 @@
---
date: 2026-11-07
branch: fix/public-dns-wan-a-record
title: external-dns v0.23 ignored every alpha-prefixed annotation — pin annotationPrefix back
---
## What changed
`platform/externaldns/helmrelease.yaml`:
- adds `values.annotationPrefix: "external-dns.alpha.kubernetes.io/"`. The chart renders it as
`--annotation-prefix=`.
- tightens the chart range from `<1.24.0` to `>=1.23.0 <1.24.0`, because that flag needs app
>= 0.22.
- adds a comment block explaining why.
**Root cause.** Renovate PR #129 (merged 12:12:00Z) moved the chart from `>=1.21.0 <1.22.0` to
`<1.24.0`. Helm deployed chart 1.23.0 at 12:13:08Z, which moved the app from v0.21.0 to v0.23.0.
The upstream v0.22.0 release notes (#6424) say: "The default annotation prefix is now
`external-dns.kubernetes.io/` with no fallback". Every external-dns annotation in this repo uses
`external-dns.alpha.kubernetes.io/`. At 12:13:12Z v0.23.0 therefore:
- **ignored the Gateway's `target`.** Every public name flipped from `CNAME
wan.home.keithjackson.dev` to `A 192.168.2.20`, an RFC1918 address served in public DNS.
- **ignored `controller: none`.** The internal names jackett, pihole and semaphore were published.
That breaks R-DNS-03.
- **ignored the L4 Services' `hostname`.** The unifi and crafty-mc A records were deleted.
cloudflare-ddns (#123, v1.18.0) is not involved. It logs only `IPv4-enabled domains:
wan.home.keithjackson.dev` and has made no changes.
**Why pin the prefix instead of migrating the annotations.** The pin is one value in one file, and
that file is not on the sensitive-path list. It restores the exact v0.21 reading of `target`,
`controller` and `hostname` together. A migration would touch `catalog/render.py`, every
`apps/wave-*` file, the hand-written platform manifests, and the conformance goldens and checks.
`conformance/**` needs an operator override. And a migration half-applied in either direction
reproduces this incident.
I rejected `--enable-legacy-annotation-prefix`. Upstream calls it a migration aid that will be
removed.
**Follow-up.** The L0 guard that the tree's annotation prefix matches this value is a separate PR
(`feat/l0-externaldns-annotation-prefix`), because it lives in `conformance/**`. A full migration
to the GA prefix remains open. It must change the annotations and this value in one commit.
## What I verified
- **Live HelmRelease history.** `1.23.0 0.23.0 lastDeployed 2026-11-07T12:13:08Z deployed`, and
before that `1.21.1 0.21.0 ... superseded`. The Deployment image is `external-dns:v0.23.0`. The
startup config logs `AnnotationPrefix:external-dns.kubernetes.io/
EnableLegacyAnnotationPrefix:false`.
- **external-dns log, 12:13:12Z:**
- `action=CREATE ... type=A` and `a-` TXT for gitea, crafty, the apex, headlamp, auth, jackett,
nextcloud, keycloak, grafana, seerr, plex, pihole and semaphore.
- `action=DELETE ... type=CNAME` and `cname-` TXT for all of those except jackett, pihole and
semaphore.
- `action=DELETE ... type=A` for unifi and crafty-mc.
- **Before the upgrade.** Loki covers about 11-05 13:00Z to 12:13Z. The old pod logged no `Changing
record` line in that window. Its only error was one Cloudflare 502 on 11-05.
- **Gateway.** `gateway/shared` still carries `external-dns.alpha.kubernetes.io/target:
wan.home.keithjackson.dev`.
- **Public DNS.** At the authority (betty.ns.cloudflare.com), 1.1.1.1 and 8.8.8.8:
- plex, gitea, auth, the apex, jackett, pihole and semaphore answer A `192.168.2.20`.
- unifi and crafty-mc have no record.
- `wan.` answers A `136.51.115.252`.
- There is no AAAA for any of these names.
- The `a-plex` and `a-jackett` TXT records read `external-dns/owner=prod`.
- **Upstream v0.23.0 source.**
- `controller/execute.go` calls `annotations.SetAnnotationPrefix(cfg.AnnotationPrefix)`.
- `SetAnnotationPrefix` rebuilds `TargetKey`, `ControllerKey` and `HostnameKey`.
- `source/gateway.go` still reads `TargetsFromTargetAnnotation(gw.Annotations)`.
- Chart 1.23.0 `templates/deployment.yaml` renders `--annotation-prefix={{
.Values.annotationPrefix }}`.
- **Phone.** Pi-hole 192.168.2.23 answers correctly: A `192.168.2.20` with TTL 60, and AAAA
NOERROR with no data. The router 192.168.2.1 answers `gitea.` with NOERROR and 0 answers, because
it strips RFC1918 results. The workstation uses only `nameserver 192.168.2.23` and has
`disable_ipv6 = 1`.
- **Toolbox gate.** See the PR description.
## What I did NOT verify
- **The fix live.** This depends on Flux applying it. Expected on the first reconcile, with
`policy: sync` and owner `prod`, all of these records being prod-owned:
- DELETE the A record and create CNAME `wan.` for the ten public names.
- DELETE the A and TXT records for jackett, pihole and semaphore.
- CREATE A for unifi (.21) and crafty-mc (.22).
This is UNVERIFIED until someone reads the external-dns log and checks with `dig
@betty.ns.cloudflare.com`. The A-to-CNAME type swap went through cleanly on 2026-10-03, and the
reverse went through today, so I expect it to converge. If it does not, see the manual list in
the PR.
- **That the chart's schema accepts `annotationPrefix` under Flux.** I did not render the
HelmRelease with helm here. The gate checks YAML and policy, not the chart render.
- **The phone.** Not investigated, by decision. Likely cause: the phone bypasses Pi-hole on the
client side. Candidates are IPv6 RA/RDNSS DNS from the router, Android Private DNS and iCloud
Private Relay. Answers it gets that way are rebind-stripped, or point to the WAN IP, which needs
hairpin NAT. Hairpin looked absent on 2026-10-03 (a self-signed cert on WAN:443). This is
router- or client-side, not in-repo.
- The AAAA records the operator saw on 8.8.8.8. I did not reproduce them.
- The cilium (#104) and pihole (#109) bumps were not examined beyond Pi-hole answering correctly.
+139
View File
@@ -0,0 +1,139 @@
---
date: 2026-11-08
branch: feat/cloudflare-tunnel
title: Cloudflare Tunnel connector, Phase 1 — the component only, isolated, with zero change to live traffic
---
## What changed
Ingress is moving to a Cloudflare Tunnel because the GFiber router cannot hairpin and hands LAN
clients Google's IPv6 DNS, so split DNS keeps breaking. This is **Phase 1: the connector only.**
external-dns, the Gateway annotations, Pi-hole and CoreDNS are untouched. A later change, by a
different agent, moves DNS.
**Operator decisions (2026-11-08), recorded here as the source for the guard:**
1. Public hostnames stay `*.home.keithjackson.dev` and the apex `home.keithjackson.dev`. The
operator enables Cloudflare ACM (ordered edge cert or Total TLS) for edge certificates.
2. **Plex stays OFF the tunnel.** No plex hostname goes through cloudflared (`plex`, `plex-vip` → 404).
3. **jackett, pihole and semaphore stay LAN/VPN-only**, never via the tunnel. The same goes for the
non-HTTP L4 VIPs **unifi** and **crafty-mc**. After the PM's read-only audit, the other three
catalog `exposure: internal` apps, **qbittorrent, radarr and sonarr**, are on the same list. All
six carry the external-dns `controller: none` route annotation live.
4. **The Kubernetes API stays local-only.** It must never be reachable through the tunnel.
5. Pi-hole keeps serving the LAN. Split DNS gets trimmed later, by another agent.
6. **Headlamp is LAN/VPN-only** (operator decision, relayed by the PM later the same day). It is
permanently in the 404 list and excluded from the egress backends. Its catalog/route exposure is
deliberately **not** changed here; the Phase 2 DNS-flip PR does that.
**Files:**
- `platform/cloudflared/` (new):
- `config.yaml` is the locally-managed tunnel config, projected through a content-hashed
`configMapGenerator`, so edits roll the pods. It has explicit `http_status:404` rules for
plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp,
unifi and crafty-mc. Then the apex and `*.${BASE_DOMAIN}` go to
`https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost: true`. Last
comes a catch-all 404. The tunnel UUID is a **literal** in this file (`REPLACE-ME-TUNNEL-UUID`
until enrolment), not a cluster-var, so the Phase 2 PR can match it without touching `clusters/**`.
- `cloudflared-credentials.sops.yaml` is SOPS-encrypted and holds `REPLACE-ME` placeholders, with
a `.example` hand-off sibling. I encrypted the placeholder (not a credential) in the toolbox
with the root `.sops.yaml` recipient.
- `deployment.yaml` runs 2 replicas with a hard `topologySpreadConstraints` across nodes. The image
is `cloudflare/cloudflared:2026.10.7@sha256:4955524a…` (tag + digest, Renovate `kubernetes`
manager). It runs as non-root 65532, with a read-only root fs, all capabilities dropped and no
SA token. Requests are 10m/64Mi and limits 500m/256Mi. Readiness uses `/ready`. Liveness uses
`/healthcheck`, so an edge outage cannot restart-loop the pods. The pods are labelled
`egress-class: internet`. A PDB sets `minAvailable: 1`.
- `networkpolicy.yaml`:
- A namespace `default-deny` with DNS only. It has a **single owner**: the namespace is not
`app-namespace=true`, so Kyverno's generate rule never writes it.
- A Prometheus scrape allow on :2000.
- The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443.
It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`,
minus the LAN-only namespaces.**
- `servicemonitor.yaml` (headless metrics Service + ServiceMonitor) and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
ship with `alerts.promtool-test.yaml`, and `README.md` documents the component.
- `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`,
**`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`.
- `clusters/{dev,prod}/kustomization.yaml` each gain a SOPS `decryption` patch for `cloudflared`.
- `conformance/tests/l0-static/cloudflared-ingress-rules/` (new, whole-repo scope) asserts several
things:
- Every rule targets either 404 or exactly the Gateway Service. The Service name is derived from
`platform/gateway/gateway.yaml`, and API markers are named explicitly.
- The excluded hosts 404 before the first Gateway rule, the Gateway rules use `matchSNItoHost`,
there is no `noTLSVerify`, and a catch-all closes the list.
- The CNP has no kube-apiserver, host, remote-node, cluster or `all` entity, no CIDR, and the full
NotIn list. The pods carry egress-class internet and have no SA token.
- The Flux tier is isolated: wait:false, not in `platform/kustomization.yaml`, nothing
`dependsOn: cloudflared`, and both overlays decrypt it.
- **Mutation self-test:** an added API rule, plex moved below the wildcard, and a dropped
catch-all must each be flagged.
- `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` gains an additive
`cloudflared_` scrape owner, with a networkpolicy companion, in the same shape as the kured entry.
- `runbooks/cloudflare-tunnel.md` (new) is registered in `runbooks/kustomization.yaml` and the
README index.
**Design choices:**
- **Isolation (the merge-before-credentials question).** The PR is safe to merge with placeholders.
The Deployment CrashLoops, but in its own `wait:false` Kustomization that nothing depends on. Its
Ready condition tracks the apply, not pod health, so neither `platform`, `platform-config` nor any
apps wave can stall. There is no Certificate or HelmRelease in the path for health checks to hang
on. Health is reported by the two alerts instead. `CloudflaredTunnelDown` **will fire from merge
until enrolment**. That is honest, and it is `warning` because nothing routes through the tunnel
yet. Raise it to critical in the DNS-flip change.
- **Egress to backends, not `{}`.** Under Cilium 1.20 (CVE-2025-30162), the Gateway's Envoy
re-checks the source pod's egress against the resolved backend. An egress rule that names only
the Gateway would therefore make every tunnelled request `403 Access denied`
(decisions/2026-10-30-gitea-admit-buildkit.md; catalog/render.py, `r_internet_egress`). The
gitea-actions `- {}` pattern would also work, but it admits kube-apiserver and host, which
decision 4 rules out. The namespace-label selector names every Gateway backend and nothing else.
**Backends need no ingress change.** Envoy reaches them as `reserved:ingress`, and the 2026-10-30
measurement showed the identity check sits on the source's egress: gitea-actions is in none of
Gitea's ingress allows and still reaches it. So no app's `inClusterClients` was touched.
- **The alerts cannot self-silence.** Both expressions use `or on() vector(0)`, so absent series
(a broken scrape, zero pods, KSM gone) fire. The promtool tests pin the absent case and the
working-then-stale re-fire case.
## What I verified
- **The Gateway Service, measured live (read-only, `k3s-prod`, 2026-11-08).** It is
`gateway/cilium-gateway-shared`: type LoadBalancer, ClusterIP `10.43.225.79`, EXTERNAL-IP
`192.168.2.20`, ports 80/443, no selector, labelled `gateway.networking.k8s.io/gateway-name=shared`.
It belongs to Gateway `gateway/shared` (class `cilium`).
- **The namespace selector, measured live.** Cilium identities carry the label
`k8s:io.cilium.k8s.namespace.labels.gateway-access` (read from a gitea identity). Every namespace
with an HTTPRoute (crafty, gitea, headlamp, homepage, jackett, keycloak, monitoring, nextcloud,
oauth2-proxy, pihole, plex, qbittorrent, radarr, runbooks, seerr, semaphore, sonarr) is
`gateway-access=shared`.
- **Nothing exists yet.** Namespace `cloudflared` does not exist on prod. **cloudflared is not in
the pinned toolbox**, so the runbook runs the pinned image under podman instead.
- The image tag and digest came from Docker Hub's API: `2026.10.7` = `latest` =
`sha256:4955524a…c9b9c5ba`, published 2026-10-30.
- The credentials Secret is encrypted: `stringData.credentials.json` is `ENC[…]`, with the age
recipient from the root `.sops.yaml`.
- The new L0 check passes on the host (47 assertions, mutation self-test included). The toolbox
gate result is reported in the PR description. I ran it after committing.
## What I did NOT verify
- **UNVERIFIED: nothing ran in a cluster.** No pod has started, and no tunnel exists. I have not
observed placeholder CrashLoop behaviour itself. I expect cloudflared to treat
`REPLACE-ME-TUNNEL-UUID` as a tunnel name, find no origin cert, and exit, but whatever it does,
the pods are not Ready and the tier is non-gating.
- **UNVERIFIED: the CVE-2025-30162 path for cloudflared specifically.** The selector form was
checked against live identity labels, but no tunnelled request has been made. The runbook's
step 6 `kubectl debug` probe exists to prove "real backend, no 403" from the pod's own identity.
- **UNVERIFIED: whether the `ingress` entity rule's `toPorts: 443` restriction holds at Envoy.**
buildkit's working rule has no port restriction. If the canary shows drops, removing the port
restriction there is the first thing to try.
- The metric name `cloudflared_tunnel_ha_connections` and the `/ready` / `/healthcheck` endpoints
come from my knowledge of cloudflared, **not measured** against 2026.10.7.
- The Cloudflare edge egress is `world` + ports, not Cloudflare CIDRs. I did not narrow it blind.
- Egress `world:443` "for the API" follows the brief. I have not confirmed that a credentials-file
tunnel needs it at all.
- The Cloudflare-side settings (ACM, Total TLS / ordered cert, Bot Fight Mode, BIC/WAF, WebSockets)
are documented for the operator but not inspected. I have no Cloudflare access.
- The headlamp, qbittorrent, radarr and sonarr catalog exposures are deliberately unchanged
(Phase 2).
@@ -0,0 +1,59 @@
---
date: 2026-11-08
branch: fix/cloudflared-creds-permission
title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
---
## What changed
- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
`root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
`couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
`fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
Still non-root, `runAsNonRoot: true`, unchanged otherwise.
- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
(runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
runAsUser unset must each be flagged. Chainsaw step description updated to match.
- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
has none), then `cloudflared tunnel cleanup`/`delete <old>`.
- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
operator is rotating the credential (it was exposed in a transcript) in a separate PR.
## What I verified
- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
`{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
`fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
`User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
`container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
- Toolbox gate: see the PR description for the rc and skipped stages.
## What I did NOT verify
- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
file mode inside the running container was not observed (pods crash before an exec is possible,
and exec would be an operator action anyway). The readability claim rests on the kubelet's
documented fsGroup behaviour and the identical, working keycloak reconciler.
- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
the operator is rotating it.
- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
syntax is from the CLI's documented commands, not run here.
- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.
@@ -0,0 +1,77 @@
---
date: 2026-11-08
branch: feat/dns-via-cloudflare-tunnel
title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard
---
## What changed
One atomic change, so the conformance guard lands with the DNS flip.
- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target`
is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was
`wan.${BASE_DOMAIN}`). Comment block rewritten.
- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because
v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The
public-target comment block rewritten, including the prefix-regression risk.
- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`:
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They
are the only `service`-source publishers in the tree and live (measured).
- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml`
re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The
render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none`
on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files
were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched.
- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`.
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and
register prose updated; new assertion (9):
(9a) Gateway target == `<tunnel: from platform/cloudflared/config.yaml>.cfargotunnel.com`, and
`--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied:
"false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the
tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set
{unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept
consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose
updated.
- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets
`cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails.
- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`,
`platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md`
(Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation
now also swaps the Gateway UUID; rollback path).
## What I verified
- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from
`gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations;
`annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the
one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a
parsed `"false"` annotation overrides it.
- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation
confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails.
- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target
back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex
golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c).
`cloudflared-ingress-rules` still passes; `render.py --selftest` PASS.
- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry
`controller: none`; plex and headlamp do not (yet); the only Services with an external-dns
hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently
`wan.home.keithjackson.dev`.
- Toolbox gate result: see the PR description (rc and skipped stages quoted there).
## What I did NOT verify
- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs
to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched.
Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`.
- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs`
is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it.
- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through
Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate,
but Cloudflare bot/WAF settings on that path are untested.
- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class),
`target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal
names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it
cannot catch a controller that ignores the flag. No runtime alert exists for it.
- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR.
- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges;
`platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME.
@@ -0,0 +1,104 @@
---
date: 2026-11-08
branch: feat/gate-toolbox-container
title: The merge gate runs inside the CI-built toolbox image, pinned by digest — no per-run tool install
---
## What changed
`.gitea/workflows/gate.yml`, job `gate` (job name, the gate step, its env and its
`|| [ "$?" -eq 3 ]` tolerance are unchanged, so the required context `gate / gate (pull_request)`
is unchanged):
- **`container:`** `gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e…ef81` — the
image `images.yaml` published from `dd94101` (run 789; it is today's `main` and `sha-dd941016478c`
tag). The same image `infra/devcontainer/dev.sh` runs, so local and CI gate on the identical
toolchain. A `# renovate-digest: image=… tag=main` marker sits on the line above, the repo's
existing convention; `infra/gitea/registry-retention.sh` reads the same marker, so this digest
is protected from pruning.
- **`actions/checkout` replaced by a plain-git step.** The toolbox has no Node (measured:
`command -v node` → missing), and `actions/checkout` is a Node action. Chosen over adding Node to
the toolbox because that would need two sequential PRs (Containerfile, then a publish, then the
pin) and would put a JS runtime into the operator toolbox only to run one action. The step
reproduces what checkout@v7 `fetch-depth: 0` actually ran here (read from runs 490 and 495):
every branch to `refs/remotes/origin/*`, tags, plus the event SHA; detached for a PR,
`-B <branch>` for a push; asserts `HEAD == github.sha`. The job token goes to that one fetch via
`GIT_CONFIG_*` env and is **not** persisted in `.git/config` (nothing later needs network git).
It sets **`umask 0022`** first: the PR's first CI run (run 500) failed L0
`prod-baremetal-not-dev-surface` because act runs steps with umask 0000, every checked-out
directory was world-writable, and Ansible (present in the toolbox, absent from the catthehacker
image) then ignored `infra/ansible/ansible.cfg`. Reproduced locally in the pinned image: clone
under umask 0000 → that check fails with the same message; under 0022 → `check OK`. Side
effect worth knowing: the Ansible-driven parts of that check now actually execute in CI.
- **"Install pinned CI toolchain" step removed**, replaced by **"Toolchain is the one
ci/versions.env pins"**: every `*_VERSION` in `ci/versions.env` must equal the image's
`/etc/dev-toolchain-versions`, else `::error::DRIFT` and the job fails. Without this, dropping
the installer would have made `ci/versions.env` an unenforced claim for everything except
promtool (which `alert-rules-unit-tested` already asserts).
- Header comments rewritten for the new job image (pull auth, cold-pull size, no-Node, why
`install-tools.sh` stays).
`renovate.json5`: the `renovate-digest` custom manager now also scans `.gitea/workflows/gate.yml`;
the `github-actions` manager is disabled for this repo's own images (same reason the `kubernetes`
manager already was: it would see a digest-only image and look up `latest`).
Comment/message wording only: `ci/gate.sh` (header + the strict-skip error text), `ci/lib.sh`,
`ci/README.md`, `.gitea/workflows/README.md` — they said the CI toolchain comes from
`install-tools.sh`.
`.gitea/workflows/scripts/install-tools.sh` is **kept**: `merge-gate.yaml` (dormant blueprint)
calls it, and `conformance/tests/l0-static/alert-rules-unit-tested/check.sh` asserts its promtool
wiring (not touched — another agent owns `conformance/`).
**Consequence to know:** every toolbox digest bump is now a Renovate PR touching `.gitea/**`,
i.e. a sensitive path needing an operator `/approve-override`. That is arguably correct (a new
toolchain changes what green means — and `ci/versions.env` was already sensitive), but it is a
new recurring operator touch.
## What I verified
- Registry pull auth: the Home-Network org is `public`; anonymous `/v2/token` → manifest `200`,
`docker-content-digest: sha256:83a3501e…ef81` for both `main` and `sha-dd941016478c`. So no
`container: credentials:` is needed. (The Actions token could not have been used anyway — no
package scope, per images.yaml's header.)
- Image: 466,912,274 bytes compressed across 6 layers (manifest sum), 1,541,728,125 bytes
unpacked (`podman image inspect`). Local cold pull 10.4s on the workstation.
- Image tool versions equal `ci/versions.env` for all 8 pins (kyverno 1.19.0, kyverno-json 0.0.3,
chainsaw 0.2.12, yamllint 1.35.1, kubectl 1.36.4, yq 4.44.6, promtool 3.14.0, check-jsonschema
0.29.4) — both via `/etc/dev-toolchain-versions` and by running each binary's `--version`.
`sleep`, `tail`, `git` 2.56.0, `bash`, `python3`, `jq`, `curl` present (act's entrypoint is
`/bin/sleep 10800`).
- The new checkout script, run in the pinned image against the public repo (auth header disabled
for the test only): PR ref `refs/pull/143/head` → detached at `60e3a24`, 1018 commits; push ref
`refs/heads/main` → `On branch main` at `e8d2666`; `origin/main` present in both. The
toolchain-drift step passes on the real `ci/versions.env` and exits 1 naming both tools when
yamllint/promtool pins are altered.
- Renovate marker regex (copied from renovate.json5) matches gate.yml exactly once, capturing
depName/tag/digest.
- LOCAL GATE, run **in the pinned CI image itself** (`IMAGE=<the digest above>` through
`dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding the main checkout's `.git` read-only),
on commit `9537ca2`: **rc 3**; spec-immutability, decisions-log (78 entries), lint, l0,
render-golden, l1, approval-policy (N/A locally — not a PR event) all pass; **gitea-settings
the only stage not run** (no `SETTINGS_API_TOKEN` locally); L2/L3 conformance skipped as
usual. rc 3 is not green.
- **CI, this PR's own runs** (Gitea API `/actions/runs/{id}/jobs` + job logs):
- run 500 (`514ed5d`, cold DinD cache: `Image exists? false`, image pulled): job 151s; set-up
incl. pull **9s**, checkout <1s, toolchain check 1s (all 8 pins `ok`), gate 140s → BLOCKED by
L0 `prod-baremetal-not-dev-surface` (umask, fixed above) + approval-policy.
- run 503 (`27d603f`, warm: `Image exists? true`): job **140s**; set-up 1s, checkout 1s,
toolchain check 0s, gate 137s. Every stage passes — l0 (incl. prod-baremetal-not-dev-surface),
lint, decisions-log, render-golden, l1, **gitea-settings** — except **approval-policy**,
the expected sensitive-path block. spec-immutability skipped (expected, no spec repo in CI).
- baseline before (runs 494/495, push on main): job 180s/179s; set-up 1–2s, checkout 1s,
**install 11s**, gate 164–165s. The install step is gone; the remaining ~25s gate-step delta
is NOT claimed as caused by this change (different base commits, one sample each).
- Re-gated locally on `27d603f` in the pinned image: rc 3, gitea-settings the only not-run stage.
## What I did NOT verify
- Renovate actually opening a digest-bump PR for gate.yml — only the regex was tested, not a
Renovate run.
- Push-to-main behaviour in CI (only the PR event runs before merge).
- Whether the gate-step speedup (165s → 137s) is real or run-to-run variance.
- That the cold-pull cost stays ~9s under a busier node or a bigger digest delta; one sample.
@@ -0,0 +1,74 @@
---
date: 2026-11-08
branch: fix/keycloak-operator-crds
title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
---
## What changed
**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
CRDs, not two (GitHub contents API, `?ref=26.8.2`):
| file | at 26.0.6 | at 26.8.2 |
|---|---|---|
| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
stayed up.
**Fix.**
- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
custom.regex manager matches all five — checked by running its regex over the file). Header
comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
`kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
installing the CRDs is upstream's supported shape.
- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
"the release URLs". `renovate.json5`: same wording fix in a comment only.
## What I verified
- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
still 450Mi/450Mi.
- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
`keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
`keycloak`.
- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
- Gate: see the PR body for the exact rc and stages of the run against the final commit.
## What I did NOT verify
- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
observed failing; another start-up failure behind them is possible.
- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
a week.
- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
`RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
one. Back up / confirm the latest CNPG backup first if that matters.
- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
`from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
Keycloak" list) — not re-checked here; PR #121 merged without it.
- `catalog/keycloak.yaml`'s documentation-only image tag was not updated.
@@ -0,0 +1,60 @@
---
date: 2026-11-08
branch: fix/keycloak-servicemonitor-owner
title: One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config)
---
## What changed
- `platform/keycloak/keycloak-instance.yaml`: added `spec.serviceMonitor.enabled: false`.
The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (`keycloak/keycloak`),
whenever `metrics-enabled=true` and the ServiceMonitor CRD exists. `serviceMonitor.enabled` defaults
to `true` in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version).
That is the same name and namespace as the repo's Flux-owned `platform/keycloak/servicemonitor.yaml`.
After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned
`409 Conflict` and it ran out of retries. The CR then sat at `Ready=False`/`HasErrors=True`, and
`platform-config` (`wait: true`, which health-checks every object it applies, this CR included)
blocked everything that `dependsOn` it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and
pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks.
- `platform/keycloak/servicemonitor.yaml`: a comment saying it is the sole owner.
- **Why ours is kept:** these are written against it: the L0 `servicemonitor-ports-resolve`
allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`, the L2 `keycloak-monitored`
chainsaw test, the README's ownership table, and the dashboard's `up{namespace="keycloak"}`.
The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the same
target: port name `management` (9000 on `keycloak-service`), path `/metrics`, interval 30s,
selector = the operator's instance labels. It offers nothing ours lacks, so having two owners
would add a fight and gain nothing.
## What I verified
- Live, read-only, `k3s-prod` ~06:25Z: the CRD schema field `spec.serviceMonitor.enabled`
(`default: true`) exists in both served versions. The live `keycloak` ServiceMonitor has exactly
**one** managedFields manager, `kustomize-controller` (Apply, 2026-09-05), **no ownerReferences**,
and labels `kustomize.toolkit.fluxcd.io/name=platform-config`, `release=kube-prometheus-stack`. So
the operator's apply never landed, and nothing on the object ties it to the Keycloak CR.
- The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent
informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING`
(`app.kubernetes.io/managed-by=keycloak-operator`). Our object does not carry that label and has
no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it.
This conclusion comes from reading the code. It was not observed.
- The other operator dependents in ns `keycloak` do not collide with anything: `keycloak-service`,
`keycloak-discovery`, the StatefulSet `keycloak`, `keycloak-initial-admin`, and
`keycloak-network-policy` (owner Keycloak, no Flux label). The repo declares none of these names.
It only *references* `keycloak-service` by name (HTTPRoute, reconcilers). There are no PDBs in the
namespace, and the Ingress is already `enabled: false`. The only NetworkPolicy in the namespace is
the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is
present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner.
Left at its default.
- Local gate: see the PR body. It records the rc and stages.
## What I did NOT verify
- That the CR reaches `Ready=True`/`HasErrors=False` after merge. The status also says
"Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears
once the workflow stops erroring. It is not proven.
- That the operator stops issuing the PATCH once `enabled: false` is applied. Watch the operator
logs for 409.
- That `platform-config` and its downstream Kustomizations go Ready on the new revision, and that
Prometheus keeps exactly one `serviceMonitor/keycloak/keycloak/0` pool with targets up.
- The operator's actual label set for `DEFAULT_LABELS_AS_STRING`. I took it from the observed labels
on operator-created objects, not from `Constants.java`.
+116
View File
@@ -0,0 +1,116 @@
---
date: 2026-11-08
branch: feat/parallel-l0-checks
title: L0 offline checks, mutation suites and yamllint run in bounded parallel pools (gate 184s -> 32s locally)
---
## What changed
The gate's wall time was dominated by serial loops over independent subprocesses. Three of
them now run in bounded pools; none of them changes a verdict, an exit code, or the order of
anything printed.
- `conformance/lib/run_l1_offline.py` — the per-check loop (L0 and L1 offline checks) runs in
a `ThreadPoolExecutor` of `bash <check>` children. `--jobs N` / env `CONF_OFFLINE_JOBS`,
default `min(8, usable CPUs)` (`sched_getaffinity`, so a cgroup cpuset is respected);
`--jobs 1` is the old serial runner. Results are consumed in SUBMISSION order, so `cases`,
the JUnit, the step summary and the 0/1/3 aggregation are built from exactly the same
ordered list as before. Additive output only: one `[offline] running N check(s), jobs=…`
line, and a trailing `[N.Ns]` wall time on each per-check status line. Children get
`stdin=/dev/null` (parallel children must not share the runner's stdin).
- `conformance/tests/l0-static/keycloak-{passkey,client-secret}-reconciler/mutations.py` —
pass 1 (serial) checks every anchor and builds each mutated source, exactly as before;
pass 2 runs the `--selftest` children in a pool (env `MUTATION_JOBS`, default
`min(8, CPUs)`, `1` = serial). Verdicts are consumed in `MUTATIONS` order. A timeout
(`TimeoutExpired`) still propagates and fails the check; queued mutations are cancelled.
An invalid `MUTATION_JOBS` exits 2 (fails the check, never skips it).
- **Nested-pool budget.** When the outer runner is parallel it exports
`MUTATION_JOBS = max(2, min(8, (cpus - jobs + 2) // 2))` unless the caller already set
one. Reason: only those two checks fan out; the other `jobs-2` workers are single
processes, so the two suites split the CPUs those leave free. Peak processes ≈
`(jobs-2) + 2*inner` ≤ cpus whenever cpus ≥ jobs+2 (24 CPUs: 6 + 16 = 22), and ~1.25–1.5×
cpus on a small runner (4 CPUs: 2 + 4 = 6). The floor of 2 is deliberate: a 1-wide inner
pool would make the 40-mutation passkey suite (~85s serial) the critical path of the stage.
- `ci/lint.sh` — yamllint runs over `LINT_JOBS` (default `min(8, nproc)`) CONTIGUOUS slices
of the sorted file list concurrently; each slice's output is buffered to its own file and
printed in slice order, so the text is the same as one serial call and a `file` header is
never separated from its `line:col` findings. Stage rc = the highest slice rc; a slice that
leaves no rc file counts as 1. Invalid `LINT_JOBS` exits 2.
### Parallel-safety audit (all 25 `l0-static/*/check.sh`, 3 `l1-rendering/*/render-check.sh`)
Grepped every check for fixed `/tmp` paths, `_artifacts`/`ARTIFACTS`/`COVERAGE_OUT`,
`write_text`/`open(…,"w")`, `mktemp`/`mkdtemp`/`TemporaryDirectory`, sockets/ports, `git`
writes and `run.sh` invocations, and read what each writer writes to:
- **Scratch state: all private.** Every writer uses its own `mktemp -d` / `mkdtemp` /
`TemporaryDirectory` (placeholder-guard-coupling, spec-immutability-enforced,
whole-repo-invariants-wired, auditability-proofs-producible, alert-rules-unit-tested and its
`platform/monitoring/tests/run.sh`, internal-services-publish-no-public-dns,
oauth2-proxy-{api-routes-per-app,cookie-names-distinct}, both keycloak reconcilers, the three
L1 render checks). No fixed `/tmp` path anywhere.
- **Git writes:** only spec-immutability-enforced, and only `git init/add/commit` inside its
own `mktemp` repo. Everything else is `git ls-files`/`ls-tree`/`log`/`rev-parse` (read-only).
- **Ports:** the reconciler selftests bind `127.0.0.1:0` (kernel-assigned); the passkey
selftest also dials `127.0.0.1:1` as a deliberate refused connection. No fixed listener.
- **Ansible** (dev-nodes-provisioned-as-code): `--syntax-check`/`--list-hosts` only,
`retry_files_enabled = False`; its local tmp is per-process.
- **The one shared writer:** auditability-proofs-producible runs
`./run.sh --coverage-only --requirements …`, which REWRITES
`conformance/_artifacts/coverage/coverage.{json,md}` (it inherits `CONF_ARTIFACTS`). That was
already true serially. No other offline check reads or writes that directory (the only
reader is `tests/meta/harness-selfcheck`, not an offline check), and the outer runner
writes coverage BEFORE and the JUnit AFTER the pool, so the final state is the same as
serial. Not serialised; left as is and recorded here. `lib/status.py` (run by that check)
reads `_artifacts/report.junit.xml` as an advisory — written only after the pool, unchanged.
- whole-repo-invariants-wired calls `run.sh --print-selection` (writes nothing) and the
runner itself against synthetic dirs in its own `mktemp`.
Result: no check needed fixing or serialising. A new check that needs shared state must say
so; the runner's docstring now states that contract.
## What I verified
All in the pinned toolbox (`dev.sh run`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding a ro
mount of the main checkout's `.git`; spec repo resolved as a sibling, spec-immutability
`pass`), host 24 CPUs, every `dev.sh` call under `flock`.
- **Gate, before** (origin/main `1a33814`, detached scratch worktree + spec mount):
`GATE_RC=3`, in-container wall **183.9s**; `l0` stage **168.3s**, `lint` **7.8s**.
"Stages that did not run: gitea-settings" (plus the expected on-base
touched-set-not-determinable note, since main has no branch commits).
- **Gate, after** (this branch, commit `a775082`): `GATE_RC=3`, in-container wall **32.1s**;
`l0` **21.1s**, `lint` **2.4s**, `l1` 2.3s. "Stages that did not run: gitea-settings" — the
only not-run stage. **rc 3, not green.** Caveat: the branch run selected 23 L0 checks (the
touched-requirement scoping dropped dev-nodes-provisioned-as-code and
placeholder-guard-coupling), the main run 25.
- **Same-tree L0 comparison** (all 25 l0 dirs, same tree, same container):
`CONF_OFFLINE_JOBS=1 MUTATION_JOBS=1` **164.0s / 165.3s** vs default (jobs=8,
MUTATION_JOBS=8) **19.2s / 18.7s**. JUnit `cmp`-identical, step summary `cmp`-identical,
stderr identical except the timing suffixes and the JUnit path in the last line.
- **Mutation suites, host:** passkey 83.2s → 10.5s (`MUTATION_JOBS=8`), client-secret
11.2s → 2.0s; output of `MUTATION_JOBS=1` and `=8` byte-identical to the pre-change script's,
rc 0 all round.
- **Falsification (not committed):** in a scratch worktree at `a775082`, made the passkey
mutation "the protected-subtree screen is never called" a no-op (replacement == anchor).
Host: `MUTATION_JOBS=1` and `=8` both rc 1, output byte-identical (with a second, broken
anchor also injected: both reported). Toolbox: runner rc 1 both serial and parallel,
`L0 FAIL keycloak-passkey-reconciler … 1 of 40 mutation(s) not proven … SURVIVED`; full
gate **rc 1**, `stage l0: FAIL`, `MERGE GATE: BLOCKED`. Reverted; scratch worktree removed.
- **lint.sh, host scratch repo** (31 YAML files, two broken): `LINT_JOBS=1/4/8` all rc 1,
stdout identical, findings named `y/f07.yaml` / `2:1 … key-duplicates` etc.; clean tree rc 0;
`LINT_JOBS=x` rc 2.
## What I did NOT verify
- **CI timings at the time of writing.** The PR's CI run is measured after push and reported
on the PR / handoff message, not here. The CI runner's CPU count is UNVERIFIED — the new
`[offline] running … jobs=… cpus=…` line in the CI log is what will show it. On a small
runner the speedup will be smaller than locally.
- **approval-policy** will be red on CI: `ci/**` and `conformance/**` are sensitive paths and
need the operator's review. I did not and must not override it.
- Flakiness under load: the selftests have no wall-clock assertions I found, and 4 toolbox
runs + several host runs were all stable, but that is a handful of runs, not a soak.
- The coverage files under `conformance/_artifacts/coverage/` end in the same state as before
only because a single check rewrites them; I did not change that pre-existing quirk.
- L2/L3 conformance: not run (no cluster in CI, as before).
Loaded 100 of 204 files, more files were not shown because too many files have changed in this diff. Show more