Commit Graph
149 Commits
Author SHA1 Message Date
jakman85 0b0b5abc8e Merge pull request 'feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools' (#149) from feat/parallel-l0-checks into main
gate / gate (push) Successful in 39s
gate / conformance (push) Skipped
Reviewed-on: #149
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:53:42 +00:00
jakman85 6e2e353bcc Merge pull request 'feat(ci): run the merge gate inside the CI-built toolbox image, pinned by digest' (#147) from feat/gate-toolbox-container into main
gate / gate (push) Successful in 2m33s
gate / conformance (push) Skipped
Reviewed-on: #147
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:51:00 +00:00
jakman85 045d81c7a2 Merge pull request 'feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs' (#146) from feat/runner-capacity-2 into main
gate / gate (push) Canceled after 1m15s
gate / conformance (push) Canceled after 0s
Reviewed-on: #146
2026-11-08 06:47:05 +00:00
jakman85 7be693bf37 Merge pull request 'chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6' (#136) from renovate/quay.io-oauth2-proxy-oauth2-proxy-7.x into main
gate / gate (push) Canceled after 1m32s
gate / conformance (push) Canceled after 0s
Reviewed-on: #136
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:45:31 +00:00
jakman85andClaude Opus 5.5 89f48e4a5f docs(decisions): handoff for parallel L0 checks; runner usage names --jobs
gate / gate (pull_request) Successful in 59s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:34:48 -06:00
jakman85andClaude Opus 5.5 64c5617ddd docs(decisions): record CI runs 500/503 for gate-toolbox-container
gate / gate (pull_request) Successful in 2m20s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:26:27 -06:00
jakman85andClaude Opus 5.5 35ba875380 fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)
gate / gate (pull_request) Successful in 3m2s
gate / conformance (pull_request) Skipped
The 26.x Keycloak operator creates its own ServiceMonitor named after the
CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled
defaults true. That collides with the Flux-owned platform/keycloak/
servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR
stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls
every downstream Kustomization.

Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are
written against it; the operator's scrapes the same port 9000 /metrics) and
set spec.serviceMonitor.enabled: false.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:25:45 -06:00
jakman85andClaude Opus 5.5 27d603f726 fix(ci): umask 0022 in the git checkout step — act's 0000 made Ansible ignore ansible.cfg
gate / gate (pull_request) Failing after 2m20s
gate / conformance (pull_request) Skipped
Run 500 (this PR's first CI run) failed L0 prod-baremetal-not-dev-surface: act runs steps with
umask 0000, the checkout was world-writable, and Ansible (now present, via the toolbox) refuses
a world-writable directory as an ansible.cfg source. Reproduced and fixed locally in the pinned
image.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:20:38 -06:00
jakman85 1a33814522 Merge pull request 'chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds' (#145) from chore/renovate-config-cleanup into main
gate / gate (push) Successful in 2m59s
gate / conformance (push) Skipped
Reviewed-on: #145
2026-11-08 06:17:59 +00:00
jakman85andClaude Opus 5.5 514ed5d5a4 docs(decisions): record local gate result for gate-toolbox-container
gate / gate (pull_request) Failing after 2m31s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:15:38 -06:00
jakman85andClaude Opus 5.5 9537ca2ddb feat(ci): run the merge gate inside the CI-built toolbox image, pinned by digest
The gate job now runs in gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e...
(the image images.yaml publishes from infra/devcontainer/Containerfile) instead of
catthehacker/ubuntu:act-22.04 plus a per-run download of yamllint, check-jsonschema, kyverno,
promtool and chainsaw. Local (dev.sh) and CI now gate on the identical toolchain.

- actions/checkout (Node) replaced by a plain-git step with fetch-depth-0 semantics; the
  toolbox has no Node. The job token is not persisted into .git/config.
- The install step is replaced by a check that every ci/versions.env pin equals the image's
  /etc/dev-toolchain-versions, so the pins stay enforced.
- Renovate follows the digest through a `# renovate-digest:` marker; github-actions manager
  disabled for own images.
- install-tools.sh kept (merge-gate.yaml blueprint + alert-rules-unit-tested check use it).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:12:03 -06:00
jakman85andClaude Opus 5.5 b548224db8 feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Skipped
The in-cluster runner ran at the default capacity 1 with no config file, so
gate / images / renovate runs on different refs queued behind each other
(29-759s waits; run 472 waited 759s behind a renovate burst).

- ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via
  CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault()
  applies the same defaults to a partial file as to no file; the
  generate-config example is deliberately not used (its labels would override
  the registered ones, force_pull differs from the default).
- Pod-template config-revision annotation so ConfigMap-only edits can roll it.
- dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi.
  Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs.
- runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1).

apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the
renderer only places it in a wave), so it is the source, not generated output.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:09:08 -06:00
jakman85andClaude Opus 5.5 759a0e51c8 chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
- Replace the deprecated top-level `docker` key with the packageRule Renovate's
  own migration produced (matchCategories docker, enabled true), keeping every
  comment. Supersedes #139, which dropped all comment lines.
- Add gitea/runner (act_runner's upstream rename) to the blast-radius rule.
- rangeStrategy bump for flux/helm `>=A <B` ranges: the default `auto` resolves
  to `replace`, which turned `>=1.12.0 <1.13.0` into `<1.14.0` (#131, #130,
  #115). In-range `bump` updates are disabled; Flux adopts those unaided.
- Restore headlamp's lower bound: `<0.47.0` -> `>=0.46.0 <0.47.0` (live 0.46.0).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:05:49 -06:00
jakman85andClaude Opus 5.5 ea30ba876c fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)
gate / gate (pull_request) Successful in 3m5s
gate / conformance (pull_request) Skipped
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2
ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new
operator fails to start its keycloaksamlclients informer (Not Found) and
CrashLoops, holding Flux `platform` in health checks.

- add both CRD URLs with the same renovate marker/depName
- patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was
  OOMKilled at 421/450Mi, 154 restarts)
- README/renovate comments: Renovate cannot add CRDs a release introduces

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:00:01 -06:00
jakman85andClaude Opus 5.5 3d7ecbcfe9 fix(conformance): regenerate L1 render goldens for oauth2-proxy v7.15.6
gate / gate (pull_request) Successful in 3m0s
gate / conformance (pull_request) Skipped
PR #136 bumps the renderer's OAUTH2_PROXY_IMAGE; the container-workload
and egress-class (vpn + internet) goldens still pinned v7.8.1. Regenerated
with the check scripts' own invocation; each diff is the image tag only.
Also updates the crafty README's current-pin row.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 23:58:29 -06:00
jakman85 048254bfaf Merge pull request 'feat(gitea): upgrade 1.27.3 -> 28.3.0' (#142) from feat/gitea-28 into main
gate / gate (push) Successful in 2m59s
gate / conformance (push) Skipped
Reviewed-on: #142
2026-11-08 05:41:34 +00:00
jakman85andClaude Opus 5.5 60e3a24652 feat(devcontainer): bake velero CLI into the toolbox, pinned to prod server v1.18.2
gate / gate (pull_request) Successful in 3m1s
gate / conformance (pull_request) Skipped
images / images (pull_request) Successful in 44s
The velero CLI and server are version-coupled, so VELERO_VERSION follows the
server running on prod (docker.io/velero/velero:v1.18.2, chart 12.2.3,
verified 2026-11-07). It has no renovate marker on purpose: bump it together
with the chart's appVersion. The verify stage asserts that
`velero version --client-only` reports the pin. The build arg reaches CI
through build-image.sh --print-build-args, so no .gitea/ or ci/ change is
needed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 23:18:21 -06:00
jakman85andClaude Opus 5.5 8a9baa92bc feat(gitea): upgrade 1.27.3 -> 28.3.0
gate / gate (pull_request) Successful in 3m13s
gate / conformance (pull_request) Skipped
Gitea dropped the "1." prefix; 28.x is the successor of 1.27.
- image 28.3.0, update.range >=28.0.0 <29.0.0, Renovate ceiling <29.0.0
- drop GITEA__server__DOMAIN (no longer read in 28, ROOT_URL is the source)
- declare RUN_RETENTION_DAYS=400, EGRESS_MODE=lax (no allow list: the OIDC
  login source is not gated by the egress policy in 28.3.0), and
  DISABLE_REGISTRATION=true (28's new default)
- document autoRollbackOnFailedHealth as inert (no ImagePolicy is rendered);
  rollback is a PVC restore because the DB migration is one-way

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 22:55:06 -06:00
jakman85andClaude Opus 5.5 9c51f60a35 feat(conformance): L0 guard that the external-dns annotationPrefix matches the tree
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
Add assertion (8) to internal-services-publish-no-public-dns. The external-dns
HelmRelease must declare annotationPrefix explicitly, and every
external-dns annotation in apps/, platform/, runbooks/ and catalog/render.py
must use that prefix. The v0.23.0 default-prefix change on 2026-11-07
ignored the controller: none, target and hostname annotations while
assertions (1)-(3) stayed green.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 17:56:12 -06:00
jakman85andClaude Opus 5.5 267446c68f fix(dns): pin external-dns annotationPrefix to the alpha prefix
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
external-dns v0.22 changed its default annotation prefix to
external-dns.kubernetes.io/ with no fallback. Renovate #129 moved the chart
to 1.23.0 (app v0.23.0), and at 12:13Z it ignored every alpha-prefixed
annotation in this repo: the Gateway target (public names went to A
192.168.2.20), controller: none (jackett, pihole and semaphore got
published), and the L4 Service hostnames (unifi and crafty-mc were deleted).

Set annotationPrefix: external-dns.alpha.kubernetes.io/ to restore the v0.21
reading. Floor the chart at 1.23.0, because the flag needs app >= 0.22.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 17:50:13 -06:00
pm-agent e8e9de8611 feat: PM-Agent also need to look at PR comments
gate / gate (pull_request) Successful in 2m58s
images / images (pull_request) Successful in 1m6s
gate / conformance (pull_request) Has been skipped
From Gitea issue #116.

Authored by the PM agent's worker; the merge gate is the approver. Refs #116.

Agent: pm-agent (issue #116, mode revise)
2026-11-07 06:02:58 +00:00
pm-agent 2fdb95393c feat: PM-Agent also need to look at PR comments
gate / gate (pull_request) Successful in 2m55s
images / images (pull_request) Successful in 1m10s
gate / conformance (pull_request) Has been skipped
From Gitea issue #116.

Authored by the PM agent's worker; the merge gate is the approver. Refs #116.

Agent: pm-agent (issue #116, mode implement)
2026-11-07 05:18:24 +00:00
jakman85andClaude Opus 5.5 19097e4d4f chore(pm-agent): pin agent-runner to the run-394 digest (466f63e, PR #99), bump config-generation
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
Dispatcher image and PM_WORKER_IMAGE both move to
sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the
registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 12:17:00 -05:00
jakman85andClaude Opus 5.5 664ee82b9a fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS
gate / gate (pull_request) Successful in 2m56s
images / images (pull_request) Successful in 13s
gate / conformance (pull_request) Has been skipped
`claude --print --output-format json` reports its own failures (API error, usage limit,
error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s
stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31.
describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails,
redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged)
makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 12:04:09 -05:00
jakman85andClaude Opus 5.5 0d3e51acc4 chore(pm-agent): turn PM_DRY_RUN off (operator-approved), bump config-generation to 4
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry,
which admission requires, and the dispatcher's dry-run triage of #97 was read and approved.
PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the
configmap comments, the README and the plan's follow-up 3 is updated to match.

Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding
present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both
egress CNPs Valid.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:45:52 -05:00
jakman85andClaude Opus 5.5 6a192ede2a chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule
refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry),
and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read
from the published layer).

Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's
dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The
renovate-digest markers are unchanged, so later bumps remain Renovate's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:22:52 -05:00
jakman85 ddd04d460e Merge pull request 'feat(pm-agent): pull credential + admission rule 7, first published digest pinned, dry-run first start, token scopes fixed' (#61) from feat/pm-agent-registry-pull-secret into main
gate / gate (push) Successful in 2m53s
images / images (push) Successful in 15s
gate / conformance (push) Has been skipped
Reviewed-on: #61
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 15:46:54 +00:00
jakman85 553d027f7c Merge branch 'main' into feat/pm-agent-registry-pull-secret
gate / gate (pull_request) Successful in 2m58s
images / images (pull_request) Successful in 34s
gate / conformance (pull_request) Has been skipped
2026-10-31 15:43:51 +00:00
jakman85andClaude Opus 5.5 183bca6897 feat(images): pin pihole, jackett and plex to the digests already running; Renovate follows them
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
pihole/pihole (no tag)               -> 2026.10.0               sha256:d3413471...
lscr.io/linuxserver/jackett:latest   -> v0.24.2865-ls7          sha256:4fcf3cc2...
plexinc/pms-docker:latest            -> 1.43.4.10903-e5521bd8c  sha256:e0ab2739...

Each tag was resolved against its registry and matched the pods' imageID, so the image is
unchanged (the pod template string does change, so Flux rolls the three workloads).

renovate.json5: regex versioning for jackett (-lsN) and plex (build-hex); jackett is looked
up on ghcr.io via overridePackageName, because lscr.io is capped at dockerMaxPages=20 pages
and jackett's 32k tags put the running one at #31,980 (dry run proposed nothing without it);
jackett weekly (13 builds in 12 days); pihole deliberately unheld, year rollover = major.
Stale "publishes only latest" prose fixed in renovate.json5 and three runbooks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 10:38:07 -05:00
jakman85 74ba90c0b3 Merge branch 'main' into fix/renovate-hold-auth-and-self
gate / gate (pull_request) Successful in 2m55s
gate / conformance (pull_request) Has been skipped
2026-10-30 08:14:15 +00:00
jakman85andClaude Opus 5.5 8ed4e30201 docs(decisions): handoff for feat/renovate-track-keycloak-operator
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 03:06:56 -05:00
jakman85andClaude Opus 5.5 df96be98bf fix(renovate): never auto-merge the oauth2-proxy sidecar or Renovate's own version
gate / gate (pull_request) Successful in 2m51s
gate / conformance (pull_request) Has been skipped
Operator decision. Both are held at every update type and labelled needs-review:

- quay.io/oauth2-proxy/oauth2-proxy (render.py constant, rendered sidecars, crafty). Since v7.13.0
  oauth2-proxy validates refreshed OIDC sessions against the access token, every sidecar uses
  --cookie-refresh, and that path is untested against this Keycloak.
- ghcr.io/renovatebot/renovate (and renovatebot/github-action if ever reintroduced). Nothing
  re-validates renovate.json5 against the new version before it takes effect.

The validator passes on 44.167.0. Evaluated with Renovate's own packageRules engine: exactly 2 of 29
branch rows change (automerge true -> false, +needs-review); the other 27 are identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:54:51 -05:00
jakman85andClaude Opus 5.5 d9571fc6d8 fix(renovate): edit catalog sources, track CNPG imageName, honour update.range, guard real-run auto-merges
gate / gate (pull_request) Successful in 2m55s
gate / conformance (pull_request) Has been skipped
Audit of the first completed dry runs (353/354) plus a debug-level local reproduction:

- App-image updates edited only the generated apps/wave-*. A ServiceDefinition has no apiVersion,
  so the kubernetes manager extracts nothing from catalog/, and every such PR would fail
  render-golden (reproduced: GOLDEN DRIFT). New regex managers for catalog source.image and
  render.py OAUTH2_PROXY_IMAGE put the source and generated edits in one branch (render-golden
  rc 0, reproduced).
- CNPG Cluster imageName was untracked (3 Clusters), while the grant Job was tracked. New manager.
- allowedVersions mirror catalog update.range: this drops gitea 28.x, qbittorrent 20.x, radarr 6,
  nextcloud 35 and postgresql 18.
- Never auto-merge: gitea, act_runner, postgresql (blast radius), flux range widening, python
  minors. Majors wait for a Dependency Dashboard tick.
- velero plugin RC: semver versioning (v1.14.3-rc.2 -> v1.14.4 now proposed).
- flux docker duplicates disabled (the "Cannot find replaceString" line). Two false comments
  corrected.

Local dry run: regex deps 7 -> 20, branches 33 -> 29. The validator passes on 44.167.0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:35:33 -05:00
jakman85andClaude Opus 5.5 0c6ee4921d fix(renovate): document and probe the five token scopes Renovate needs; lint daemon-bound actions
gate / gate (pull_request) Successful in 2m54s
gate / conformance (pull_request) Has been skipped
Run 350 cleared platform init (/user 200, /version 200, /repos 200), then stopped at
GET /api/v1/orgs/Home-Network with Gitea's own 403:
  token does not have at least one of required scope(s), required=[read:organization],
  token scope=read:package,write:repository,read:user
The workflow's header listed only those three scopes, and the operator minted exactly that.
Renovate's initRepo -> isOrg needs read:organization, and Gitea files /issues and /labels under
the issue scope, which the dashboard and PR labels need: write:issue.

- renovate.yaml: corrects the scope list (read:user, read:organization, write:repository,
  write:issue, read:package), with the call each serves. A new preflight step probes them with
  GETs, the token passed to curl on stdin, and names a missing scope instead of Renovate's bare
  "Authentication failure".
- ci/lint.sh: rejects workflow `uses:` lines that need a Docker daemon (docker://,
  renovatebot/github-action, docker/*-action), the class fixed in #84.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:55:39 -05:00
jakman85andClaude Opus 5.5 365e43cdea fix(renovate): run Renovate as the job container; the action needs a Docker daemon jobs do not get
gate / gate (pull_request) Successful in 2m52s
gate / conformance (pull_request) Has been skipped
renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:

  failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory

The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.

Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:23:36 -05:00
jakman85andClaude Opus 5.5 7f0cbc7ba2 feat(pm-agent): pin the first published agent-runner digest, start dry, and fix the token scopes
gate / gate (pull_request) Failing after 2m56s
images / images (pull_request) Successful in 32s
gate / conformance (pull_request) Has been skipped
Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git
blockers to going live go away:

- deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3...,
  tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged.
- PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not
  just cautious: the run-340 image's baked worker template predates imagePullSecrets, so
  admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps.
- Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under
  read:user. The documented list was missing it. Now read:user, write:repository,
  write:issue, read:package everywhere.
- Org-package read documented as a team (Packages=Read, no repos), not just a scope.
- sops is toolbox-only; the .examples say so and give the in-place edit.
- runbook: key-name check no longer emits Secret values before the pipe.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 01:16:21 -05:00
jakman85andClaude Opus 5 a7eadd4054 feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.

pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.

ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.

The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.

kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.

The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-30 01:09:39 -05:00
jakman85 262c6e4a97 Merge branch 'main' into feat/keycloak-passkey-reconcile
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-10-30 05:56:00 +00:00
jakman85andClaude Opus 5.5 d2c4bd5194 fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 00:38:54 -05:00
jakman85andClaude Opus 5.5 04eb38584a fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal
gate / gate (pull_request) Successful in 1m58s
images / images (pull_request) Failing after 7s
gate / conformance (pull_request) Has been skipped
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 f65c12b395 feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.

- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
  digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
  read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
  :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
  No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
  `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
  the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
  targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
  Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 8f35234fcc fix(images): name the measured runner blocker; the job container gets no Docker daemon
PR #81's first images run failed in 6s: act_runner 0.2.11 forces docker_host '-' for a tcp://
DOCKER_HOST, and gitea/act drops every bind when valid_volumes is empty. Giving jobs the
privileged DinD daemon is node-root for any workflow author, so it is left to the operator;
options are in the handoff entry. The driver's failure now names the cause.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 e539109830 feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.

- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
  builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
  sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
  (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
  manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
  current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
  Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5.5 da1b36eb64 fix(renovate): pin renovate-version — the action's default (39) rejects renovate.json5
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:49:38 -05:00
jakman85andClaude Opus 5.5 6bb8af1d70 docs(decisions): handoff for the passkey reconciler review fixes
gate / gate (pull_request) Failing after 2m55s
gate / conformance (pull_request) Has been skipped
Also states in bind_browser_flow()'s docstring that it is reached only when git's
bind-browser-flow is "true".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 21:45:40 -05:00
jakman85andClaude Opus 5 f5931944fa feat(keycloak): reconcile the passkey realm config via GitOps, the browser flow blue/green
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.

A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.

  pass 1  webAuthnPolicyPasswordless* -- strict read-modify-write of
          GET/PUT /admin/realms/homelab, readback-verified.
  pass 2  webauthn-register-passwordless -- strict read-modify-write of ONE
          literally-named alias; enabled, defaultAction false so enrolment stays
          opt-in.
  pass 3  the browser flow -- BLUE/GREEN.

PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:

  1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
     shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
     mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
     and zero user-visible effect.
  2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
     conditional-OTP config CONTENTS -- before anything is bound, so the
     regression that once 403'd every platform-admin is caught BEFORE it bites.
  3. BIND with ONE atomic PUT setting browserFlow.
  4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
     flow survives as the rollback target.

The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.

Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.

Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.

Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.

Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-29 21:22:08 -05:00
jakman85andClaude Opus 5.5 aacf361bb8 fix(renovate): the workflow still named renovate.json, so every run since #79 died with ENOENT
gate / gate (pull_request) Failing after 1m58s
gate / conformance (pull_request) Has been skipped
#79 renamed the config to renovate.json5 and left configurationFile pointing at the old
name. Runs 315-322 (2026-10-22..29) all failed in ~1s before Renovate started.

Also add a ci/lint.sh assertion that every workflow configurationFile: resolves to a real
path — valid YAML, so yamllint could not catch it, and the gate stayed green throughout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 20:58:15 -05:00
jakman85 1301af9a4c Merge pull request 'fix(renovate): the config was INVALID, so the first manual run errored out' (#79) from fix/renovate-invalid-config into main
gate / gate (push) Successful in 1m55s
gate / conformance (push) Has been skipped
renovate / renovate (push) Failing after 9s
Reviewed-on: #79
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:29:34 +00:00
jakman85 42ce001d71 Merge pull request 'fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR' (#77) from fix/renovate-cron-window into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #77
Reviewed-by: Keith Jackson <[email protected]>
2026-10-18 14:28:59 +00:00
jakman85andClaude Opus 5 0d4bd7297d fix(renovate): the cron fired outside Renovate's own window, so it would never open a PR
gate / conformance (pull_request) Has been skipped
gate / gate (pull_request) Successful in 1m55s
.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.

  03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
  window     ->  03:00-06:00 local   =>  NOT INSIDE (3h45m early)

The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.

Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
  EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
  EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.

Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.

STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:12:28 -05:00