docs(decisions): record CI runs 500/503 for gate-toolbox-container

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.gitea/workflows/README.md

@@ -165,7 +165,8 @@ Cost: the job shares the single act_runner slot with the gate, but the build its
   merge-gate.yaml            the pipeline (lint → l0 → l1 → conformance → decision)
   images.yaml                build/verify/publish this repo's container images (see above)
   scripts/
-    install-tools.sh         installs the pinned tools per stage (from ci/versions.env)
+    install-tools.sh         installs the pinned tools per stage (from ci/versions.env) — used by the
+                             dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
     decide.sh                the R-TEST-03 merge decision over job results
     images.sh                the image pipeline's driver (what to build, base, push, digest)
   README.md                  this file

.gitea/workflows/gate.yml

@@ -3,8 +3,10 @@
 # This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
 # Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
 # exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
-# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
-# below for why the script, not `make`, is invoked here). It runs every stage in order —
+# (R-TEST-02) — and since 2026-11-08 it runs INSIDE the same image: the job `container:` is the
+# CI-built devcontainer toolbox (see "THE JOB IMAGE" below), not a generic runner image that
+# downloads the pinned tools on every run. (See further below for why the script, not `make`,
+# is invoked here.) It runs every stage in order —
 # spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
 # R-TEST-03 decision, printing
 # "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
@@ -17,7 +19,8 @@
 # kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
 # environment it is:
 #   * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
-#     installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
+#     baked into the job image and checked against ci/versions.env two steps up.
+#     GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
 #     "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
 #   * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
 #     not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
@@ -41,6 +44,32 @@
 # same script the laptop runs; the extra pre-step is a read-only git-diff check and is
 # purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
 # is retained dormant in merge-gate.yaml.
+#
+# THE JOB IMAGE (2026-11-08, feat/gate-toolbox-container). The gate job runs in
+# gitea.home.keithjackson.dev/home-network/toolbox, pinned BY DIGEST — the image
+# .gitea/workflows/images.yaml builds from infra/devcontainer/Containerfile, verifies, and
+# publishes. It is the SAME image `infra/devcontainer/dev.sh` runs, so a local gate and the CI
+# gate now execute the identical toolchain (R-TEST-02), not two installs that agree by intent.
+# Before this, the job ran in ghcr.io/catthehacker/ubuntu:act-22.04 and spent ~11s of every run
+# re-downloading yamllint, check-jsonschema, kyverno, promtool and chainsaw
+# (.gitea/workflows/scripts/install-tools.sh l0 + l1; measured on runs 494/495).
+#   * PINNED BY DIGEST, MOVED BY RENOVATE. The `# renovate-digest:` marker above the image line
+#     is the repo-wide convention (renovate.json5, "This repo's own CI-published images"):
+#     Renovate follows the `main` tag and opens a digest-bump PR, whose OWN gate run then
+#     executes on the new image — the bump tests itself. infra/gitea/registry-retention.sh reads
+#     the same marker, so the pinned digest is never pruned. Never hand-edit it to a tag.
+#   * PULL AUTH: none needed. The Home-Network org is public and the registry issues an
+#     anonymous pull token (measured 2026-11-08: anonymous /v2/token -> manifest 200), so the
+#     runner's DinD pulls it like any public image. If the org is ever made private, add
+#     `credentials:` here with a read:package token secret — the automatic Actions token has no
+#     package scope at /v2/token (see images.yaml's header).
+#   * COLD PULL: ~467 MB compressed (1.54 GB unpacked). The DinD store is an emptyDir, so the
+#     first job after a runner-pod restart or a digest bump pays one pull; every other run finds
+#     it cached (forcePull=false), exactly as it already did for the catthehacker image.
+#   * NO NODE in the toolbox, so `actions/checkout` (a Node action) cannot run here. The checkout
+#     step is plain git instead — see its comment for how it keeps fetch-depth-0 semantics.
+#   * install-tools.sh is NOT deleted: merge-gate.yaml (the dormant blueprint) still calls it,
+#     and conformance/tests/l0-static/alert-rules-unit-tested asserts its promtool wiring.
 ---
 name: gate
 
@@ -78,16 +107,76 @@ jobs:
   # Single required status check: the static merge gate.
   gate:
     runs-on: ubuntu-latest
+    # The CI-built toolbox — the same image dev.sh runs. See "THE JOB IMAGE" in the header.
+    container:
+      # renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
+      image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
     steps:
-      - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0        # full history so ci/touched-requirements.sh can diff vs main
-
-      - name: Install pinned CI toolchain (ci/versions.env)
+      # Plain git, because actions/checkout is a Node action and the toolbox has no Node. It
+      # reproduces what actions/checkout@v7 with `fetch-depth: 0` did here (read from its logged
+      # commands on runs 490/495): full history of every branch + tags, so
+      # ci/touched-requirements.sh can diff against origin/<base>; the exact event commit
+      # checked out (detached for a PR, as `-B <branch>` for a push). One difference, on
+      # purpose: the job token is passed to the single fetch through GIT_CONFIG_* environment
+      # variables and is NOT persisted into .git/config — nothing after this step needs
+      # authenticated git (ci/ and conformance/ make no network git calls), so there is no
+      # reason to leave a credential in the workspace for the gate's scripts to read.
+      - name: Checkout (git, full history)
+        env:
+          GIT_TOKEN: ${{ github.token }}
+          SERVER_URL: ${{ github.server_url }}
+          REPOSITORY: ${{ github.repository }}
+          SHA: ${{ github.sha }}
+          REF: ${{ github.ref }}
         run: |
-          ./.gitea/workflows/scripts/install-tools.sh l0
-          ./.gitea/workflows/scripts/install-tools.sh l1
+          set -euo pipefail
+          # act runs steps with umask 0000, so without this every checked-out directory is
+          # world-writable — and Ansible then IGNORES infra/ansible/ansible.cfg as a source
+          # ("Ansible is being run in a world writable directory"), which made the L0 check
+          # prod-baremetal-not-dev-surface fail on this PR's first CI run (run 500). The
+          # toolbox has Ansible, so that check now drives it for real. 0022 = what a
+          # workstation clone gets (reproduced locally: 0000 fails the check, 0022 passes).
+          umask 0022
+          git config --global --add safe.directory "$PWD"
+          git init -q .
+          git remote add origin "${SERVER_URL}/${REPOSITORY}"
+          auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
+          GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
+          GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
+            git -c protocol.version=2 fetch --quiet --no-tags --prune --no-recurse-submodules origin \
+              '+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' \
+              "+${SHA}:refs/remotes/checkout/head"
+          unset auth GIT_TOKEN
+          case "$REF" in
+            refs/heads/*) git checkout -q --force -B "${REF#refs/heads/}" "$SHA" ;;
+            *)            git checkout -q --force --detach "$SHA" ;;
+          esac
+          [ "$(git rev-parse HEAD)" = "$SHA" ] || { echo "::error::HEAD is not ${SHA}" >&2; exit 1; }
+          echo "checked out $(git log -1 --format='%H %s') (ref ${REF}); $(git rev-list --count HEAD) commits of history"
+
+      # The image is the toolchain, so prove it is the PINNED one before trusting it: every
+      # *_VERSION in ci/versions.env must equal what the image recorded at build time
+      # (/etc/dev-toolchain-versions, written by infra/devcontainer/Containerfile; its `verify`
+      # stage already asserted the version-sensitive binaries report those pins). This replaces
+      # the per-run install: a drifted toolbox fails HERE, naming the tool, instead of gating with
+      # a different engine. ci/versions.env and infra/devcontainer/versions.env must move together.
+      - name: Toolchain is the one ci/versions.env pins
+        run: |
+          set -euo pipefail
+          . ci/versions.env
+          drift=0
+          for var in $(grep -oE '^[A-Z_]+_VERSION' ci/versions.env); do
+            tool="$(printf '%s' "${var%_VERSION}" | tr 'A-Z_' 'a-z-')"
+            want="${!var}"
+            got="$(sed -n "s/^${tool}=//p" /etc/dev-toolchain-versions)"
+            if [ "$got" = "$want" ]; then
+              printf 'ok     %-18s %s\n' "$tool" "$want"
+            else
+              printf '::error::DRIFT  %s: ci/versions.env pins %s, the job image has %s\n' "$tool" "$want" "${got:-nothing}"
+              drift=1
+            fi
+          done
+          [ "$drift" -eq 0 ]
 
       # R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
       # spec-immutability check, so its result is a distinct, tolerant status line in the

.gitea/workflows/images.yaml

@@ -107,7 +107,7 @@ jobs:
     timeout-minutes: 75
     steps:
       - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           fetch-depth: 0   # the per-image change detection diffs against the base commit
 

.gitea/workflows/merge-gate.yaml

@@ -60,9 +60,9 @@ jobs:
   spec-immutability:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262                 # v4                 # the implementation repo
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1                 # v7                 # the implementation repo
       - name: Check out the read-only spec (sibling path)
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           repository: ${{ vars.SPEC_REPO_SLUG }}  # e.g. Home-Network/home-network-k3s (set in Gitea)
           path: ../home-network-k3s
@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [spec-immutability]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }          # full history so touched-requirement diff works
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh lint
@@ -85,7 +85,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [lint]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l0
@@ -96,7 +96,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [l0]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -107,7 +107,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [render-golden]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -115,7 +115,7 @@ jobs:
         run: make -C ci l1
       - name: Upload L1 artifacts
         if: always()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: l1-junit-coverage
           path: |
@@ -131,7 +131,7 @@ jobs:
     needs: [l1]
     if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh conformance
@@ -142,7 +142,7 @@ jobs:
         run: make -C ci conformance
       - name: Upload L2/L3 artifacts
         if: always()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: conformance-junit-coverage
           path: |
@@ -160,7 +160,7 @@ jobs:
     needs: [lint, l0, render-golden, l1, conformance]
     if: always()
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       - name: Decide
         env:
           R_LINT: ${{ needs.lint.result }}

apps/wave-0/gitea.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: gitea
-        image: gitea/gitea:1.27.3
+        image: gitea/gitea:28.3.0
         ports:
         - containerPort: 3000
         env:
@@ -89,14 +89,18 @@ spec:
             secretKeyRef:
               name: gitea-bootstrap
               key: secret-key
-        - name: GITEA__server__DOMAIN
-          value: gitea.${BASE_DOMAIN}
         - name: GITEA__server__ROOT_URL
           value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
           value: github
+        - name: GITEA__actions__RUN_RETENTION_DAYS
+          value: '400'
+        - name: GITEA__security__EGRESS_MODE
+          value: lax
+        - name: GITEA__service__DISABLE_REGISTRATION
+          value: 'true'
         - name: GITEA_ADMIN_USERNAME
           valueFrom:
             secretKeyRef:

apps/wave-0/nextcloud-db.yaml

@@ -48,7 +48,7 @@ spec:
   # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
   # pg_basebackup from the running primary, which is neither restarted nor disconnected.
   instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
@@ -300,7 +300,7 @@ spec:
               # is already in it and it is already present on every node, so this
               # adds no new image and no new supply-chain surface. Keep the tag in
               # step with spec.imageName above when Renovate bumps it.
-              image: ghcr.io/cloudnative-pg/postgresql:16.4
+              image: ghcr.io/cloudnative-pg/postgresql:16.15
               imagePullPolicy: IfNotPresent
               command:
                 - psql

apps/wave-0/semaphore-db.yaml

@@ -49,7 +49,7 @@ spec:
   # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
   # pg_basebackup from the running primary, which is neither restarted nor disconnected.
   instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:

apps/wave-1/gitea-actions.yaml

@@ -133,7 +133,7 @@ spec:
       initContainers:
         - name: dind
           restartPolicy: Always   # <- this field is what makes it a SIDECAR
-          image: docker:27-dind
+          image: docker:29-dind
           securityContext:
             privileged: true
           env:

catalog/gitea.yaml

@@ -10,7 +10,7 @@
 name: gitea
 kind: container
 source:
-  image: gitea/gitea:1.27.3     # pinned recent stable tag (R-UPD-01); update.range governs advancement
+  image: gitea/gitea:28.3.0     # pinned recent stable tag (R-UPD-01); update.range governs advancement
 # Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
 # ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
 # without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
@@ -23,14 +23,43 @@ env:
       secretKeyRef:
         name: gitea-bootstrap
         key: secret-key
-  - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.${BASE_DOMAIN}
+  # ROOT_URL is the ONLY source of the instance domain since Gitea 28 (go-gitea/gitea#39400): [server]
+  # DOMAIN is no longer read, and the SSH domain defaults from ROOT_URL too. GITEA__server__DOMAIN was
+  # removed in the 28 upgrade; the stale DOMAIN line environment-to-ini already wrote into app.ini is
+  # inert. Correct public clone/redirect URLs for the TLS-terminated host come from this one value.
   - name: GITEA__server__ROOT_URL
     value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
     value: github                              # has internet egress; not air-gapped)
+  # Gitea 28 added run retention (go-gitea/gitea#38855): completed runs, with their jobs, logs and
+  # artifacts, are deleted by the cleanup_action_runs cron after RUN_RETENTION_DAYS. Upstream's default
+  # (400) is kept but DECLARED, so the deletion horizon is visible here rather than implicit. The spec
+  # repo states no CI-history retention requirement; 0 would mean keep forever.
+  - name: GITEA__actions__RUN_RETENTION_DAYS
+    value: "400"
+  # Gitea 28 (go-gitea/gitea#39426) puts webhooks, the OAuth2 avatar fetch and OpenID 2.0 behind an
+  # egress policy: in the default lax mode, PUBLIC targets are allowed and private/loopback/CGNAT
+  # targets need an IP or CIDR entry in [security] ALLOWED_HOST_LIST (a hostname entry is NOT enough
+  # for a private address). DECLARED lax with an EMPTY allow list, deliberately:
+  #   * the Keycloak OIDC login source is NOT gated by it. In 28.3.0 the openidConnect provider is
+  #     goth's, built with its default HTTP client (services/auth/source/oauth2/providers_openid.go);
+  #     egress.NewSecurityPolicy is used only by routers/web/auth/oauth.go (avatar fetch, and
+  #     [oauth2_client] UPDATE_AVATAR is off here) and modules/auth/openid (OpenID 2.0, unused).
+  #     So keycloak.<domain> resolving to the private Gateway address in-cluster does not matter.
+  #   * this instance has no webhooks and no mirrors (measured 2026-11-07), and 1.27 already refused
+  #     private webhook targets by default, so nothing that worked before is newly denied.
+  # Adding an entry here would only WIDEN the webhook SSRF surface. Add the narrowest CIDR (e.g. the
+  # Gateway address /32) only if a real private webhook target appears.
+  - name: GITEA__security__EGRESS_MODE
+    value: lax
+  # Gitea 28 (go-gitea/gitea#39400) disables LOCAL self-registration unless DISABLE_REGISTRATION is
+  # explicitly false; 1.27 defaulted it open. Declared true: accounts arrive via Keycloak only.
+  # OAuth2 auto-registration below is unaffected — routers/web/auth/oauth.go gates it on
+  # ALLOW_ONLY_INTERNAL_REGISTRATION and ENABLE_AUTO_REGISTRATION, not on DISABLE_REGISTRATION.
+  - name: GITEA__service__DISABLE_REGISTRATION
+    value: "true"
   - name: GITEA_ADMIN_USERNAME               # break-glass local admin creds (consumed by the postStart hook)
     valueFrom:
       secretKeyRef:
@@ -233,9 +262,14 @@ auth:
   # recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
 update:
   channel: stable               # R-UPD-01
-  range: ">=1.22.0 <2.0.0"      # stay on the 1.x stable line
+  range: ">=28.0.0 <29.0.0"     # Gitea dropped the "1." prefix at 28.0.0 (successor of 1.27); stay on 28.x
   cadence: "0 4 * * *"
-  autoRollbackOnFailedHealth: true   # R-UPD-03
+  # R-UPD-03. INERT FOR THIS SERVICE, and it must stay that way: this flag renders only an annotation
+  # on a Flux ImagePolicy, and apps/ is rendered --no-image-automation (ci/render-golden.sh), so no
+  # ImagePolicy exists and the Deployment (strategy Recreate) is never auto-reverted. That matters
+  # because a Gitea upgrade (1.27 -> 28 included) MIGRATES THE SQLITE DB ONE-WAY: an image revert onto
+  # a migrated DB is not a rollback. Roll Gitea back by RESTORING THE DATA PVC from its pre-upgrade backup.
+  autoRollbackOnFailedHealth: true
 observability:
   metrics: true
   slis:

ci/README.md

@@ -99,7 +99,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
 
 | stage | a skip there means | verdict |
 |---|---|---|
-| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
+| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is the job image — the CI-built toolbox, `.gitea/workflows/gate.yml` `container:` — so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
 | `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
 | `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
 | `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |

ci/gate.sh

@@ -46,8 +46,8 @@
 # WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
 #
 #   lint / l0 / render-golden / l1   A skip here means a PINNED tool is absent or a wired
-#     seam went unwired. On a CI runner the pinned toolchain is installed for us by
-#     .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
+#     seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
+#     CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
 #     environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
 #     it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
 #     in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
@@ -257,7 +257,7 @@ for s in lint l0 render-golden l1; do
     skip)
       if strict_skips; then
         err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
-        err "  toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
+        err "  toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
         err "  seam is on main — so this is a DEFECT, not an environment -> blocks merge"
         summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
         BLOCK=1

ci/lib.sh

@@ -97,9 +97,9 @@ cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
 # --- Skip strictness: is this a CI runner, or a workstation? -------------------
 # A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
 #
-#   CI       the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
-#            before the gate runs, and every seam is on `main`. A stage that skips here is a
-#            DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
+#   CI       the pinned toolchain IS the job image (the CI-built toolbox; gate.yml `container:`),
+#            and every seam is on `main`. A stage that skips here is a
+#            DEFECT (a tool dropped from the image, a seam that went unwired) -> BLOCK.
 #   local    a missing pinned tool is a real, ordinary local condition -> the gate reports
 #            UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
 #

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -26,7 +26,10 @@
 #       KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
 #   (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
 #       ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
-#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
+#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
+#   (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
+#       prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
+#       the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime.
 #
 # WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
 #   - That the record is gone from Cloudflare. Live read; belongs beside
@@ -90,9 +93,18 @@ CHECKS = 0
 #   ControllerValue = "dns-controller"                  -> the value that CLAIMS an object
 # source/annotations/processors.go IsControllerMismatch skips a source object when the key is
 # PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
-CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+#
+# THE PREFIX IS NOT A CONSTANT OF NATURE. Every key external-dns reads is `<annotationPrefix><name>`,
+# and the prefix is a runtime flag whose DEFAULT moved in v0.22.0 (#6424: alpha -> GA, "with no
+# fallback"). On 2026-11-07 the chart bump to v0.23.0 made every key below invisible to the running
+# controller while this check — which only asserted the keys were PRESENT — stayed green. (8) now
+# ties ANNOTATION_PREFIX to the HelmRelease's explicit `annotationPrefix`.
+ANNOTATION_PREFIX = "external-dns.alpha.kubernetes.io/"
+CONTROLLER_KEY = ANNOTATION_PREFIX + "controller"
 CONTROLLER_CLAIMED = "dns-controller"
-HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
+HOSTNAME_KEY = ANNOTATION_PREFIX + "hostname"
+# Any key that looks like an external-dns annotation, under ANY prefix variant (alpha, GA, typos).
+EXTERNAL_DNS_KEY_RE = re.compile(r"^external-dns(\.[a-z0-9-]+)*\.kubernetes\.io/")
 
 
 def check(cond, msg):
@@ -231,9 +243,7 @@ for sub in SCAN_DIRS:
             ann = md.get("annotations") or {}
             if kind == "HTTPRoute":
                 routes.append((rel, d))
-            elif kind == "Service" and any(
-                a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
-            ):
+            elif kind == "Service" and any(EXTERNAL_DNS_KEY_RE.match(a) for a in ann):
                 dns_services.append((rel, d))
 
 check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
@@ -444,6 +454,67 @@ check(
     "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
 )
 
+# =============================================================================================
+# (8) THE RUNNING CONTROLLER READS THE PREFIX THIS TREE WRITES.
+#     (a) the external-dns HelmRelease sets `values.annotationPrefix` EXPLICITLY (an upstream
+#         default is exactly what moved under us) and it equals ANNOTATION_PREFIX;
+#     (b) it does not ALSO enable `--enable-legacy-annotation-prefix` (dual-read hides a
+#         half-migration; upstream says the flag will be removed);
+#     (c) every external-dns-looking annotation key on any object under apps/ platform/ runbooks/
+#         uses ANNOTATION_PREFIX — a key under another prefix is silently ignored at runtime;
+#     (d) catalog/render.py emits no external-dns key under another prefix.
+#     Migrating to the GA prefix = change ANNOTATION_PREFIX, the HelmRelease value, the renderer
+#     and every manifest in ONE commit; this assertion fails on any partial state.
+# =============================================================================================
+HR_PATH = ROOT / "platform/externaldns/helmrelease.yaml"
+hr = next((d for d in docs_of(HR_PATH) if d.get("kind") == "HelmRelease"
+           and (d.get("metadata") or {}).get("name") == "external-dns"), None)
+check(hr is not None, f"(8) {HR_PATH.relative_to(ROOT)} holds no HelmRelease `external-dns` — "
+                      f"cannot prove which annotation prefix the controller reads")
+values = ((hr or {}).get("spec") or {}).get("values") or {}
+hr_prefix = values.get("annotationPrefix")
+check(
+    hr_prefix == ANNOTATION_PREFIX,
+    f"(8a) external-dns HelmRelease `values.annotationPrefix` is {hr_prefix!r}, but every "
+    f"external-dns annotation in this tree uses {ANNOTATION_PREFIX!r}. A mismatch (or relying on "
+    f"the upstream default, which changed in v0.22.0) makes the controller IGNORE `controller: "
+    f"none`, the Gateway `target` and Service `hostname` — internal names go public, public names "
+    f"answer RFC1918 (incident 2026-11-07).",
+)
+extra = values.get("extraArgs") or []
+extra_s = " ".join(f"{k}={v}" for k, v in extra.items()) if isinstance(extra, dict) \
+    else " ".join(map(str, extra))
+check("enable-legacy-annotation-prefix" not in extra_s and "annotation-prefix" not in extra_s,
+      "(8b) external-dns extraArgs must not set --annotation-prefix or "
+      "--enable-legacy-annotation-prefix; the prefix is declared ONLY via values.annotationPrefix "
+      "so this check can see it")
+
+foreign: list[str] = []
+seen_keys = 0
+for sub in SCAN_DIRS:
+    base = ROOT / sub
+    if not base.is_dir():
+        continue
+    for p in sorted(base.rglob("*.yaml")):
+        rel = p.relative_to(ROOT).as_posix()
+        for d in docs_of(p):
+            for k in annotations_of(d):
+                if isinstance(k, str) and EXTERNAL_DNS_KEY_RE.match(k):
+                    seen_keys += 1
+                    if not k.startswith(ANNOTATION_PREFIX):
+                        foreign.append(f"{rel}:{'/'.join(ident(d))} {k}")
+check(seen_keys > 0, "(8c) no external-dns annotation found anywhere — parse contract broken "
+                     "(the Gateway target alone guarantees at least one)")
+check(not foreign,
+      f"(8c) external-dns annotations under a prefix the controller does NOT read "
+      f"({ANNOTATION_PREFIX!r} is configured): {foreign}")
+render_src = read(CATALOG / "render.py")
+bad_render = sorted({m for m in re.findall(r"external-dns(?:\.[a-z0-9-]+)*\.kubernetes\.io/",
+                                           render_src) if m != ANNOTATION_PREFIX})
+check(not bad_render,
+      f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
+      f"reads {ANNOTATION_PREFIX!r}")
+
 # =============================================================================================
 if FAILURES:
     for f in FAILURES:

decisions/2026-11-07-devcontainer-velero-cli.md

@@ -0,0 +1,54 @@
+---
+date: 2026-11-07
+branch: feat/devcontainer-velero-cli
+title: Bake the velero CLI into the dev toolbox, pinned to the prod server version
+---
+
+## What changed
+
+- `infra/devcontainer/versions.env`: new `VELERO_VERSION=1.18.2`, with a comment explaining that
+  the CLI and server are version-coupled, so this pin follows the server and is not bumped on
+  its own. It has **no** `# renovate:` marker on purpose. When a `velero` chart bump in
+  `platform/velero/helmrelease.yaml` changes the appVersion, bump this pin in the same PR.
+- `infra/devcontainer/Containerfile`: `ARG VELERO_VERSION`; the binary comes from the
+  vmware-tanzu GitHub release tarball and is installed in the same RUN as flux/helm. There is
+  no checksum check, which matches the other tools in that file (none of them verify one).
+  `velero=` is recorded in `/etc/dev-toolchain-versions`. The verify stage now checks that
+  `velero` is on PATH, and `check velero v${VELERO_VERSION}` asserts the output of
+  `velero version --client-only`.
+- `infra/devcontainer/build-image.sh`: `VELERO_VERSION` added to `BUILD_ARGS`. CI reads that
+  list generically through `--print-build-args` (`.gitea/workflows/scripts/images.sh`), so
+  **no sensitive path (`.gitea/**`, `ci/**`) was touched**.
+- `infra/devcontainer/README.md`: added a Backups row to the tool table.
+- `runbooks/backup-failure.md`: under "First commands", a read-only
+  `velero backup describe --details` / `backup logs` example run through dev.sh.
+
+## What I verified
+
+- Live server version, read-only: `kubectl --context k3s-prod -n velero get deploy velero`
+  shows image `docker.io/velero/velero:v1.18.2`, and HelmRelease `velero` has
+  lastAttemptedRevision `12.2.3`.
+- Local build: `IMAGE=localhost/home-k3s-dev:velero-wt ./build-image.sh` returned rc 0. The
+  verify stage printed `ok velero` (on PATH), `ok velero v1.18.2` (pin check) and
+  `TOOLBOX VERIFIED`. The image was built under a separate local tag and **not pushed**.
+- Smoke test with that image against prod (read-only): `velero version` reports Client v1.18.2
+  and Server v1.18.2. `velero backup get` lists the daily backups, all `Completed` with 0 errors.
+- Recurring warning (reported only, not fixed): every daily backup since
+  `velero-daily-20261031030050` has 1 warning. Backups up to 20261030 have 0. The warning text is:
+  `gitea-actions: resource: /pods name: /act-runner-5944dfdfb8-69bvq message: /Skip pod volume
+  data error: /pod is not in the expected status, ... phase=Failed: pod is not running`.
+  It is the same pod in 20261031, 20261104 and 20261108. `gitea-actions` holds 11 `Failed`
+  pods with reason `Evicted` from ReplicaSet `act-runner-5944dfdfb8`, ten of them started
+  2026-10-30T08:26Z. The running runner is `act-runner-fc597f6b5-vsvkl`. So the warning comes from a
+  leftover evicted (`Failed`) pod that Velero still tries to back up volumes for. Cleaning up the evicted pods is an
+  operator action.
+- Gate: see the PR description for the toolbox `ci/gate.sh` rc and the skipped stages.
+
+## What I did NOT verify
+
+- The CI image build (`.gitea/workflows/images.yaml`) on this branch. It runs on push/PR and I
+  have not seen its result while writing this.
+- An arm64 build. Only amd64 was built (`TARGETARCH` defaults to amd64).
+- Any mutating velero command (`backup create`, `restore`, `delete`). None were run.
+- Why only `-69bvq` of the 11 evicted act-runner pods shows up in the warning. A likely guess is
+  that only that pod's volumes match the backup's pod-volume selection, but I did not check.

decisions/2026-11-07-gitea-28.md

@@ -0,0 +1,62 @@
+---
+date: 2026-11-07
+branch: feat/gitea-28
+title: "Gitea 1.27.3 -> 28.3.0 (upstream dropped the 1. prefix)"
+---
+
+## What changed
+
+- `catalog/gitea.yaml`: `source.image` `gitea/gitea:1.27.3` -> `gitea/gitea:28.3.0` (still the
+  newest stable tag on Docker Hub and GitHub on 2026-11-07: 28.0.0, 28.1.0, 28.2.0, 28.3.0).
+  `update.range` `">=1.22.0 <2.0.0"` -> `">=28.0.0 <29.0.0"`.
+- 28.x breaking changes handled in the env block:
+  - `GITEA__server__DOMAIN` **removed**: 28 no longer reads `[server] DOMAIN` (go-gitea/gitea#39400);
+    `ROOT_URL` was already set and is now the only source of the domain. Nothing else in the repo
+    reads the variable (the bootstrap script sets ROOT_URL/SSH_PORT only).
+  - `GITEA__actions__RUN_RETENTION_DAYS=400` **declared** (go-gitea/gitea#38855 added the 400-day
+    default; the spec repo states no CI-history retention requirement).
+  - `GITEA__security__EGRESS_MODE=lax` **declared, with no allow list.** Read from the 28.3.0
+    source: the Keycloak OIDC login source does NOT go through the new egress policy (goth's
+    openidConnect provider, default HTTP client). The policy covers webhooks, the OAuth2 avatar
+    fetch (`UPDATE_AVATAR` is off) and OpenID 2.0. A hostname entry would not even help for a
+    private address — `WithLocalNeedsIPAllow` requires an IP/CIDR entry. So the brief's
+    "Keycloak SSO breaks" premise does not hold for 28.3.0, and an allow-list entry would only
+    widen webhook SSRF reach.
+  - `GITEA__service__DISABLE_REGISTRATION=true` **declared** (28's new default; 1.27 defaulted it
+    open). OAuth2 auto-registration is gated separately and keeps working.
+- `update.autoRollbackOnFailedHealth`: kept `true`, commented as **inert**. It renders only an
+  annotation on a Flux ImagePolicy, and `apps/` is rendered `--no-image-automation`, so no
+  ImagePolicy exists. The Deployment is `Recreate` with no probes, so nothing reverts the image
+  automatically. Rollback = restore the PVC, never an image revert onto the migrated DB.
+- `apps/wave-0/gitea.yaml` re-rendered with the canonical command
+  (`render.py --all catalog -o apps --wave-layout --no-image-automation`, in the toolbox). Only
+  gitea changed.
+- `renovate.json5`: gitea `allowedVersions` `<2.0.0` -> `<29.0.0`; the description no longer calls
+  28.x a stale tag.
+
+## What I verified
+
+- Docker Hub tag list and GitHub tags: 28.3.0 is the newest 28.x stable tag.
+- Read the v28.3.0 source: `modules/egress/policy/policy.go`, `modules/egress/policies.go`, every
+  `egress.New*` caller, `services/auth/source/oauth2/providers_openid.go`,
+  `modules/setting/{security,actions,service}.go`, `routers/web/auth/{oauth,linkaccount}.go`,
+  `cmd/admin_user_list.go`, `cmd/admin_auth*.go`, the Dockerfile, and `docker/root` (the s6 setup
+  still runs `environment-to-ini`). The postStart hook's CLI is unchanged in 28.3.0: `admin user
+  list --admin` still has Username in column 2, `admin auth list` still has ID/Name in columns 1/2,
+  `add-oauth`/`update-oauth` keep the same flags, and curl and su-exec are still in the image.
+- Live Gitea (API, 2026-11-07): version 1.27.3. Zero repo, org and system webhooks, no mirrors.
+- `renovate-config-validator` at 44.167.0 (the tag renovate.yaml pins): "Config validated
+  successfully".
+- Ran the toolbox gate on the committed tree (numbers are in the PR).
+
+## What I did NOT verify
+
+- **The upgrade itself.** Nothing booted 28.3.0 against a copy of the production DB. Migration
+  time, migration success and the post-upgrade SSO login are all UNVERIFIED, and the PR's
+  post-merge checklist covers them.
+- **The age of the latest gitea-data Longhorn/off-cluster backup.** The cluster read failed from
+  the worktree (no k3s-prod context with the kubeconfig available), so the operator must confirm
+  it before merging.
+- How the Envoy Gateway handles the new `/-/ws` WebSocket. If upgrades are not forwarded, live
+  notifications fall back to polling (not breaking). UNVERIFIED.
+- Whether the L2/L3 conformance suite passes. It does not run in CI.

decisions/2026-11-07-l0-externaldns-annotation-prefix.md

@@ -0,0 +1,51 @@
+---
+date: 2026-11-07
+branch: feat/l0-externaldns-annotation-prefix
+title: L0 guard — external-dns annotationPrefix must match every external-dns annotation in the tree
+---
+
+## What changed
+
+`conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` gains assertion
+(8). The check is already whole-repo-scoped, so it runs on every gate.
+
+- **(8a)** `platform/externaldns/helmrelease.yaml` must set `values.annotationPrefix` explicitly,
+  and it must equal the check's `ANNOTATION_PREFIX` (`external-dns.alpha.kubernetes.io/`). The
+  check does not trust the upstream default, because that default is what changed.
+- **(8b)** `extraArgs` must not carry `--annotation-prefix` or `--enable-legacy-annotation-prefix`,
+  so the prefix has one declared place.
+- **(8c)** Every external-dns-looking annotation key under apps/, platform/ and runbooks/ must use
+  that prefix. The match is any `external-dns(.x)*.kubernetes.io/` key.
+- **(8d)** catalog/render.py must emit no external-dns key under any other prefix.
+
+`CONTROLLER_KEY` and `HOSTNAME_KEY` are now derived from `ANNOTATION_PREFIX`. The Service
+inventory matches any external-dns prefix variant, not only the alpha one.
+
+Why: on 2026-11-07, external-dns v0.23.0 (whose default prefix is GA) ignored every alpha
+annotation. Assertions (1) to (3) stayed green, because they prove the annotations are present,
+not that the controller reads them. See `decisions/2026-11-07-public-dns-wan-a-record.md`.
+
+This branch is stacked on `fix/public-dns-wan-a-record` (PR #140). Without that fix, (8a) fails,
+which is correct. Merge #140 first.
+
+## What I verified
+
+Run on the host (PyYAML):
+
+- the check passes on this tree: `check OK: 62 assertions passed`.
+- it **fails (8a)** with origin/main's HelmRelease, which has no `annotationPrefix`.
+- it **fails (8c)** when `platform/gateway/gateway.yaml`'s `target` key is changed to the GA
+  prefix.
+
+Toolbox gate: see the PR description.
+
+## What I did NOT verify
+
+- Whether a future chart could read the prefix from somewhere other than `values.annotationPrefix`,
+  for example a renamed chart value. (8a) would then pass while the flag silently vanished from the
+  rendered args. Rendering the chart in the gate would close that gap. Not done: it needs network
+  access or a vendored chart in L0.
+- `conformance/**` is a **sensitive path**. This PR needs the operator's approving review or
+  `/approve-override`. The agent did not post one.
+- The header of the sibling chainsaw-test.yaml still says "external-dns v0.21.0". That is cosmetic
+  and I left it alone.

decisions/2026-11-07-public-dns-wan-a-record.md

@@ -0,0 +1,99 @@
+---
+date: 2026-11-07
+branch: fix/public-dns-wan-a-record
+title: external-dns v0.23 ignored every alpha-prefixed annotation — pin annotationPrefix back
+---
+
+## What changed
+
+`platform/externaldns/helmrelease.yaml`:
+
+- adds `values.annotationPrefix: "external-dns.alpha.kubernetes.io/"`. The chart renders it as
+  `--annotation-prefix=`.
+- tightens the chart range from `<1.24.0` to `>=1.23.0 <1.24.0`, because that flag needs app
+  >= 0.22.
+- adds a comment block explaining why.
+
+**Root cause.** Renovate PR #129 (merged 12:12:00Z) moved the chart from `>=1.21.0 <1.22.0` to
+`<1.24.0`. Helm deployed chart 1.23.0 at 12:13:08Z, which moved the app from v0.21.0 to v0.23.0.
+The upstream v0.22.0 release notes (#6424) say: "The default annotation prefix is now
+`external-dns.kubernetes.io/` with no fallback". Every external-dns annotation in this repo uses
+`external-dns.alpha.kubernetes.io/`. At 12:13:12Z v0.23.0 therefore:
+
+- **ignored the Gateway's `target`.** Every public name flipped from `CNAME
+  wan.home.keithjackson.dev` to `A 192.168.2.20`, an RFC1918 address served in public DNS.
+- **ignored `controller: none`.** The internal names jackett, pihole and semaphore were published.
+  That breaks R-DNS-03.
+- **ignored the L4 Services' `hostname`.** The unifi and crafty-mc A records were deleted.
+
+cloudflare-ddns (#123, v1.18.0) is not involved. It logs only `IPv4-enabled domains:
+wan.home.keithjackson.dev` and has made no changes.
+
+**Why pin the prefix instead of migrating the annotations.** The pin is one value in one file, and
+that file is not on the sensitive-path list. It restores the exact v0.21 reading of `target`,
+`controller` and `hostname` together. A migration would touch `catalog/render.py`, every
+`apps/wave-*` file, the hand-written platform manifests, and the conformance goldens and checks.
+`conformance/**` needs an operator override. And a migration half-applied in either direction
+reproduces this incident.
+
+I rejected `--enable-legacy-annotation-prefix`. Upstream calls it a migration aid that will be
+removed.
+
+**Follow-up.** The L0 guard that the tree's annotation prefix matches this value is a separate PR
+(`feat/l0-externaldns-annotation-prefix`), because it lives in `conformance/**`. A full migration
+to the GA prefix remains open. It must change the annotations and this value in one commit.
+
+## What I verified
+
+- **Live HelmRelease history.** `1.23.0 0.23.0 lastDeployed 2026-11-07T12:13:08Z deployed`, and
+  before that `1.21.1 0.21.0 ... superseded`. The Deployment image is `external-dns:v0.23.0`. The
+  startup config logs `AnnotationPrefix:external-dns.kubernetes.io/
+  EnableLegacyAnnotationPrefix:false`.
+- **external-dns log, 12:13:12Z:**
+  - `action=CREATE ... type=A` and `a-` TXT for gitea, crafty, the apex, headlamp, auth, jackett,
+    nextcloud, keycloak, grafana, seerr, plex, pihole and semaphore.
+  - `action=DELETE ... type=CNAME` and `cname-` TXT for all of those except jackett, pihole and
+    semaphore.
+  - `action=DELETE ... type=A` for unifi and crafty-mc.
+- **Before the upgrade.** Loki covers about 11-05 13:00Z to 12:13Z. The old pod logged no `Changing
+  record` line in that window. Its only error was one Cloudflare 502 on 11-05.
+- **Gateway.** `gateway/shared` still carries `external-dns.alpha.kubernetes.io/target:
+  wan.home.keithjackson.dev`.
+- **Public DNS.** At the authority (betty.ns.cloudflare.com), 1.1.1.1 and 8.8.8.8:
+  - plex, gitea, auth, the apex, jackett, pihole and semaphore answer A `192.168.2.20`.
+  - unifi and crafty-mc have no record.
+  - `wan.` answers A `136.51.115.252`.
+  - There is no AAAA for any of these names.
+  - The `a-plex` and `a-jackett` TXT records read `external-dns/owner=prod`.
+- **Upstream v0.23.0 source.**
+  - `controller/execute.go` calls `annotations.SetAnnotationPrefix(cfg.AnnotationPrefix)`.
+  - `SetAnnotationPrefix` rebuilds `TargetKey`, `ControllerKey` and `HostnameKey`.
+  - `source/gateway.go` still reads `TargetsFromTargetAnnotation(gw.Annotations)`.
+  - Chart 1.23.0 `templates/deployment.yaml` renders `--annotation-prefix={{
+    .Values.annotationPrefix }}`.
+- **Phone.** Pi-hole 192.168.2.23 answers correctly: A `192.168.2.20` with TTL 60, and AAAA
+  NOERROR with no data. The router 192.168.2.1 answers `gitea.` with NOERROR and 0 answers, because
+  it strips RFC1918 results. The workstation uses only `nameserver 192.168.2.23` and has
+  `disable_ipv6 = 1`.
+- **Toolbox gate.** See the PR description.
+
+## What I did NOT verify
+
+- **The fix live.** This depends on Flux applying it. Expected on the first reconcile, with
+  `policy: sync` and owner `prod`, all of these records being prod-owned:
+  - DELETE the A record and create CNAME `wan.` for the ten public names.
+  - DELETE the A and TXT records for jackett, pihole and semaphore.
+  - CREATE A for unifi (.21) and crafty-mc (.22).
+  This is UNVERIFIED until someone reads the external-dns log and checks with `dig
+  @betty.ns.cloudflare.com`. The A-to-CNAME type swap went through cleanly on 2026-10-03, and the
+  reverse went through today, so I expect it to converge. If it does not, see the manual list in
+  the PR.
+- **That the chart's schema accepts `annotationPrefix` under Flux.** I did not render the
+  HelmRelease with helm here. The gate checks YAML and policy, not the chart render.
+- **The phone.** Not investigated, by decision. Likely cause: the phone bypasses Pi-hole on the
+  client side. Candidates are IPv6 RA/RDNSS DNS from the router, Android Private DNS and iCloud
+  Private Relay. Answers it gets that way are rebind-stripped, or point to the WAN IP, which needs
+  hairpin NAT. Hairpin looked absent on 2026-10-03 (a self-signed cert on WAN:443). This is
+  router- or client-side, not in-repo.
+- The AAAA records the operator saw on 8.8.8.8. I did not reproduce them.
+- The cilium (#104) and pihole (#109) bumps were not examined beyond Pi-hole answering correctly.

decisions/2026-11-08-gate-toolbox-container.md

@@ -0,0 +1,104 @@
+---
+date: 2026-11-08
+branch: feat/gate-toolbox-container
+title: The merge gate runs inside the CI-built toolbox image, pinned by digest — no per-run tool install
+---
+
+## What changed
+
+`.gitea/workflows/gate.yml`, job `gate` (job name, the gate step, its env and its
+`|| [ "$?" -eq 3 ]` tolerance are unchanged, so the required context `gate / gate (pull_request)`
+is unchanged):
+
+- **`container:`** `gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e…ef81` — the
+  image `images.yaml` published from `dd94101` (run 789; it is today's `main` and `sha-dd941016478c`
+  tag). The same image `infra/devcontainer/dev.sh` runs, so local and CI gate on the identical
+  toolchain. A `# renovate-digest: image=… tag=main` marker sits on the line above, the repo's
+  existing convention; `infra/gitea/registry-retention.sh` reads the same marker, so this digest
+  is protected from pruning.
+- **`actions/checkout` replaced by a plain-git step.** The toolbox has no Node (measured:
+  `command -v node` → missing), and `actions/checkout` is a Node action. Chosen over adding Node to
+  the toolbox because that would need two sequential PRs (Containerfile, then a publish, then the
+  pin) and would put a JS runtime into the operator toolbox only to run one action. The step
+  reproduces what checkout@v7 `fetch-depth: 0` actually ran here (read from runs 490 and 495):
+  every branch to `refs/remotes/origin/*`, tags, plus the event SHA; detached for a PR,
+  `-B <branch>` for a push; asserts `HEAD == github.sha`. The job token goes to that one fetch via
+  `GIT_CONFIG_*` env and is **not** persisted in `.git/config` (nothing later needs network git).
+  It sets **`umask 0022`** first: the PR's first CI run (run 500) failed L0
+  `prod-baremetal-not-dev-surface` because act runs steps with umask 0000, every checked-out
+  directory was world-writable, and Ansible (present in the toolbox, absent from the catthehacker
+  image) then ignored `infra/ansible/ansible.cfg`. Reproduced locally in the pinned image: clone
+  under umask 0000 → that check fails with the same message; under 0022 → `check OK`. Side
+  effect worth knowing: the Ansible-driven parts of that check now actually execute in CI.
+- **"Install pinned CI toolchain" step removed**, replaced by **"Toolchain is the one
+  ci/versions.env pins"**: every `*_VERSION` in `ci/versions.env` must equal the image's
+  `/etc/dev-toolchain-versions`, else `::error::DRIFT` and the job fails. Without this, dropping
+  the installer would have made `ci/versions.env` an unenforced claim for everything except
+  promtool (which `alert-rules-unit-tested` already asserts).
+- Header comments rewritten for the new job image (pull auth, cold-pull size, no-Node, why
+  `install-tools.sh` stays).
+
+`renovate.json5`: the `renovate-digest` custom manager now also scans `.gitea/workflows/gate.yml`;
+the `github-actions` manager is disabled for this repo's own images (same reason the `kubernetes`
+manager already was: it would see a digest-only image and look up `latest`).
+
+Comment/message wording only: `ci/gate.sh` (header + the strict-skip error text), `ci/lib.sh`,
+`ci/README.md`, `.gitea/workflows/README.md` — they said the CI toolchain comes from
+`install-tools.sh`.
+
+`.gitea/workflows/scripts/install-tools.sh` is **kept**: `merge-gate.yaml` (dormant blueprint)
+calls it, and `conformance/tests/l0-static/alert-rules-unit-tested/check.sh` asserts its promtool
+wiring (not touched — another agent owns `conformance/`).
+
+**Consequence to know:** every toolbox digest bump is now a Renovate PR touching `.gitea/**`,
+i.e. a sensitive path needing an operator `/approve-override`. That is arguably correct (a new
+toolchain changes what green means — and `ci/versions.env` was already sensitive), but it is a
+new recurring operator touch.
+
+## What I verified
+
+- Registry pull auth: the Home-Network org is `public`; anonymous `/v2/token` → manifest `200`,
+  `docker-content-digest: sha256:83a3501e…ef81` for both `main` and `sha-dd941016478c`. So no
+  `container: credentials:` is needed. (The Actions token could not have been used anyway — no
+  package scope, per images.yaml's header.)
+- Image: 466,912,274 bytes compressed across 6 layers (manifest sum), 1,541,728,125 bytes
+  unpacked (`podman image inspect`). Local cold pull 10.4s on the workstation.
+- Image tool versions equal `ci/versions.env` for all 8 pins (kyverno 1.19.0, kyverno-json 0.0.3,
+  chainsaw 0.2.12, yamllint 1.35.1, kubectl 1.36.4, yq 4.44.6, promtool 3.14.0, check-jsonschema
+  0.29.4) — both via `/etc/dev-toolchain-versions` and by running each binary's `--version`.
+  `sleep`, `tail`, `git` 2.56.0, `bash`, `python3`, `jq`, `curl` present (act's entrypoint is
+  `/bin/sleep 10800`).
+- The new checkout script, run in the pinned image against the public repo (auth header disabled
+  for the test only): PR ref `refs/pull/143/head` → detached at `60e3a24`, 1018 commits; push ref
+  `refs/heads/main` → `On branch main` at `e8d2666`; `origin/main` present in both. The
+  toolchain-drift step passes on the real `ci/versions.env` and exits 1 naming both tools when
+  yamllint/promtool pins are altered.
+- Renovate marker regex (copied from renovate.json5) matches gate.yml exactly once, capturing
+  depName/tag/digest.
+- LOCAL GATE, run **in the pinned CI image itself** (`IMAGE=<the digest above>` through
+  `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding the main checkout's `.git` read-only),
+  on commit `9537ca2`: **rc 3**; spec-immutability, decisions-log (78 entries), lint, l0,
+  render-golden, l1, approval-policy (N/A locally — not a PR event) all pass; **gitea-settings
+  the only stage not run** (no `SETTINGS_API_TOKEN` locally); L2/L3 conformance skipped as
+  usual. rc 3 is not green.
+
+- **CI, this PR's own runs** (Gitea API `/actions/runs/{id}/jobs` + job logs):
+  - run 500 (`514ed5d`, cold DinD cache: `Image exists? false`, image pulled): job 151s; set-up
+    incl. pull **9s**, checkout <1s, toolchain check 1s (all 8 pins `ok`), gate 140s → BLOCKED by
+    L0 `prod-baremetal-not-dev-surface` (umask, fixed above) + approval-policy.
+  - run 503 (`27d603f`, warm: `Image exists? true`): job **140s**; set-up 1s, checkout 1s,
+    toolchain check 0s, gate 137s. Every stage passes — l0 (incl. prod-baremetal-not-dev-surface),
+    lint, decisions-log, render-golden, l1, **gitea-settings** — except **approval-policy**,
+    the expected sensitive-path block. spec-immutability skipped (expected, no spec repo in CI).
+  - baseline before (runs 494/495, push on main): job 180s/179s; set-up 1–2s, checkout 1s,
+    **install 11s**, gate 164–165s. The install step is gone; the remaining ~25s gate-step delta
+    is NOT claimed as caused by this change (different base commits, one sample each).
+- Re-gated locally on `27d603f` in the pinned image: rc 3, gitea-settings the only not-run stage.
+
+## What I did NOT verify
+
+- Renovate actually opening a digest-bump PR for gate.yml — only the regex was tested, not a
+  Renovate run.
+- Push-to-main behaviour in CI (only the PR event runs before merge).
+- Whether the gate-step speedup (165s → 137s) is real or run-to-run variance.
+- That the cold-pull cost stays ~9s under a busier node or a bigger digest delta; one sample.

infra/devcontainer/Containerfile

@@ -30,6 +30,9 @@ ARG CHECK_JSONSCHEMA_VERSION
 # Pinned to the Prometheus version prod actually runs — see versions.env for why that lockstep is
 # the point, and for the live-cluster command that establishes it.
 ARG PROMTOOL_VERSION
+# velero: operator CLI for the backup server. Pinned to the server version prod runs — the CLI
+# and server are version-coupled; see versions.env.
+ARG VELERO_VERSION
 ARG TARGETARCH=amd64
 
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
@@ -66,6 +69,10 @@ RUN set -eux; \
     curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" -o /usr/local/bin/kubectl; chmod +x /usr/local/bin/kubectl; \
     curl -fsSL "https://get.helm.sh/helm-v${HELM_VERSION}-linux-${TARGETARCH}.tar.gz" | tar -xz -C /tmp; \
       install -m0755 "/tmp/linux-${TARGETARCH}/helm" /usr/local/bin/helm; \
+    VELERO_TARBALL="velero-v${VELERO_VERSION}-linux-${TARGETARCH}"; \
+    curl -fsSL "https://github.com/vmware-tanzu/velero/releases/download/v${VELERO_VERSION}/${VELERO_TARBALL}.tar.gz" \
+      | tar -xz -C /usr/local/bin --strip-components=1 "${VELERO_TARBALL}/velero"; \
+      chmod +x /usr/local/bin/velero; \
     rm -rf /tmp/*
 
 # Pinned conformance test toolchain (L0/L1 static gate + the L2/L3 Chainsaw runner).
@@ -93,7 +100,7 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
       echo "kyverno=${KYVERNO_VERSION}"; echo "kyverno-json=${KYVERNO_JSON_VERSION}"; \
       echo "chainsaw=${CHAINSAW_VERSION}"; echo "yamllint=${YAMLLINT_VERSION}"; \
       echo "yq=${YQ_VERSION}"; echo "check-jsonschema=${CHECK_JSONSCHEMA_VERSION}"; \
-      echo "promtool=${PROMTOOL_VERSION}"; \
+      echo "promtool=${PROMTOOL_VERSION}"; echo "velero=${VELERO_VERSION}"; \
     } > /etc/dev-toolchain-versions
 
 WORKDIR /work
@@ -112,6 +119,7 @@ ARG KUBECTL_VERSION
 ARG CHAINSAW_VERSION
 ARG KYVERNO_VERSION
 ARG SOPS_VERSION
+ARG VELERO_VERSION
 RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
 echo "--- no network (the build-time stand-in for run --network none) ---"
 if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
@@ -122,13 +130,13 @@ echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
 echo "--- every tool is on PATH ---"
 for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
          yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
-         wg qrencode; do
+         wg qrencode velero; do
   if command -v "$t" >/dev/null 2>&1; then printf "    ok      %s\n" "$t";
   else printf "    MISSING %s\n" "$t"; missing=1; fi
 done
 [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
 echo "--- version-sensitive tools report their pins ---"
-: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
+: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}" "${VELERO_VERSION:?}"
 check() { # name pin actual-output
   if printf "%s" "$3" | grep -qF "$2"; then printf "    ok      %-10s %s\n" "$1" "$2";
   else printf "    DRIFT   %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
@@ -138,6 +146,7 @@ check kubectl  "${KUBECTL_VERSION}"  "$(kubectl version --client 2>&1)"
 check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
 check kyverno  "${KYVERNO_VERSION}"  "$(kyverno version 2>&1)"
 check sops     "${SOPS_VERSION}"     "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
+check velero   "v${VELERO_VERSION}"  "$(velero version --client-only 2>&1)"
 [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
 echo "--- python can import what the renderer and the gate need ---"
 python3 -c "import yaml; print('    PyYAML', yaml.__version__)"

infra/devcontainer/README.md

@@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses.
 | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
 | GitOps / cluster | `flux`, `kubectl`, `helm` |
 | Secrets | `sops`, `age` + `age-keygen` |
+| Backups (`platform/velero`) | `velero` CLI — pinned to the **server** version prod runs (version-coupled; see `versions.env`) |
 | VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
 | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
 | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |

infra/devcontainer/build-image.sh

@@ -54,6 +54,7 @@ BUILD_ARGS=(
   "KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
   "YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
   "CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
+  "VELERO_VERSION=$VELERO_VERSION"
 )
 if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
 

infra/devcontainer/versions.env

@@ -44,4 +44,19 @@ CHECK_JSONSCHEMA_VERSION=0.29.4
 # promtool ships INSIDE the prometheus release tarball (prometheus-<v>.linux-<arch>.tar.gz), so
 # the version string here is a Prometheus release, not a separate promtool release.
 PROMTOOL_VERSION=3.14.0
+# velero — the operator CLI for the Velero backup server (platform/velero): `velero backup
+# describe <name> --details`, `velero backup logs`, `velero backup create --from-schedule`.
+#
+# THIS PIN IS NOT FREE-FLOATING: the CLI and server are VERSION-COUPLED (the CLI talks to the
+# server's CRDs and its download-request protocol; upstream supports only a matching minor), so
+# it MUST equal the Velero server version actually running on prod.
+# Verified against the live cluster on 2026-11-07:
+#   kubectl --context k3s-prod -n velero get deploy velero \
+#     -o jsonpath='{.spec.template.spec.containers[0].image}'
+#     -> docker.io/velero/velero:v1.18.2   (HelmRelease velero, chart velero-12.2.3)
+# Deliberately carries NO `# renovate:` marker: it must move WITH the server, i.e. when a
+# vmware-tanzu `velero` chart bump (platform/velero/helmrelease.yaml, range >=12 <13) changes the
+# chart's appVersion, bump this in the same PR. Never bump it on its own. The verify stage
+# asserts `velero version --client-only` reports exactly this version.
+VELERO_VERSION=1.18.2
 # Proxmox provider (bpg/proxmox) is pinned in the tofu module (infra/tofu/versions.tf), not here.

platform/externaldns/helmrelease.yaml

@@ -11,7 +11,10 @@ spec:
   chart:
     spec:
       chart: external-dns
-      version: ">=1.21.0 <1.22.0"   # k8s-1.36 catch-up (app v0.21; policy already set). Renovate maintains
+      # chart 1.23.x = app v0.23.0. Floor >=1.23.0 because `annotationPrefix` below needs app >=0.22.
+      # Renovate maintains — but READ THE APP RELEASE NOTES on every 0.x minor: 0.22 changed the
+      # annotation prefix with no fallback and silently re-published every internal name (2026-11-07).
+      version: ">=1.23.0 <1.24.0"
       sourceRef:
         kind: HelmRepository
         name: external-dns
@@ -25,6 +28,29 @@ spec:
   values:
     provider:
       name: cloudflare
+    # ---------------------------------------------------------------------------------------------
+    # ANNOTATION PREFIX — PINNED TO THE LEGACY `alpha` PREFIX (2026-11-07). Do not remove.
+    #
+    # external-dns v0.22.0 (#6424) changed the default annotation prefix from
+    # `external-dns.alpha.kubernetes.io/` to `external-dns.kubernetes.io/` "with no fallback".
+    # EVERY external-dns annotation in this repo uses the alpha prefix: the Gateway's `target`
+    # (platform/gateway/gateway.yaml), the renderer's `controller: none` on `internal` routes
+    # (catalog/render.py -> apps/wave-*), and the L4 Services' `hostname` (crafty-mc, unifi).
+    # Renovate PR #129 moved the chart range to `<1.24.0` (app v0.21.0 -> v0.23.0) and on the first
+    # reconcile (2026-11-07T12:13Z) v0.23.0, reading only the GA prefix, saw none of them:
+    #   * `target` ignored  -> every public name flipped CNAME wan.<base> -> A 192.168.2.20 (RFC1918)
+    #   * `controller: none` ignored -> internal jackett, pihole, semaphore PUBLISHED (R-DNS-03 broken)
+    #   * `hostname` ignored -> unifi and crafty-mc A records DELETED
+    # Logged as `action=CREATE ... type=A` / `action=DELETE ... type=CNAME` for every name at once.
+    #
+    # The chart's `annotationPrefix` renders `--annotation-prefix=`, which v0.23.0 applies to every
+    # key it reads (source/annotations/annotations.go SetAnnotationPrefix: target, controller,
+    # hostname, ttl, ...), restoring exactly the v0.21 behaviour. Chosen over
+    # `--enable-legacy-annotation-prefix` (a dual-read migration aid upstream says will be removed)
+    # and over migrating the annotations (renderer + goldens + conformance = sensitive paths, and a
+    # flag-day across ~20 objects). Migrating to the GA prefix is the follow-up: change every
+    # annotation AND this value in ONE commit, never one without the other.
+    annotationPrefix: "external-dns.alpha.kubernetes.io/"
     # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
     #
     # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block

platform/headlamp/helmrelease.yaml

@@ -40,7 +40,7 @@ spec:
       # values (-session-ttl + the -oidc-* args from the externalSecret; useCookie stays default-false
       # so `-oidc-use-cookie` is NOT emitted) is supported by the v0.41.0 backend. Renovate maintains
       # within 0.41.x; the appVersion continues to pin the Headlamp image.
-      version: ">=0.41.0 <0.42.0"
+      version: "<0.47.0"
       sourceRef:
         kind: HelmRepository
         name: headlamp

platform/keycloak/kustomization.yaml

@@ -28,11 +28,11 @@ resources:
   # branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
   # passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
-  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
-  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
-  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/kubernetes.yml
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml
   # --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
   - keycloak-admin-credentials.sops.yaml
   # --- OIDC client secrets (grafana + oauth2-proxy) the realm import injects via placeholders ---

platform/keycloak/postgres-cluster.yaml

@@ -41,7 +41,7 @@ spec:
   # not disconnected. (The anti-affinity change is a separate commit precisely because that one DOES
   # cause a switchover — see it for the blast radius.)
   instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes (2026-09-07).
   #
   # CNPG's default is `preferred`, which is only a scheduler HINT: the scheduler may put two

renovate.json5

@@ -142,7 +142,9 @@
       "description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
       "managerFilePatterns": [
         "/(^|/)versions\\.env$/",
-        "/(^|/)platform/pm-agent/.+\\.ya?ml$/"
+        "/(^|/)platform/pm-agent/.+\\.ya?ml$/",
+        // ADDED 2026-11-08: the merge gate's job `container:` is the toolbox, by digest.
+        "/^\\.gitea/workflows/gate\\.ya?ml$/"
       ],
       "matchStrings": [
         "#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
@@ -290,8 +292,8 @@
       "enabled": false
     },
     {
-      "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
-      "matchManagers": ["kubernetes"],
+      "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids. `github-actions` ADDED 2026-11-08 for the same reason: it extracts the gate job's digest-only `container: image:` in .gitea/workflows/gate.yml, which the custom manager already follows through its marker.",
+      "matchManagers": ["kubernetes", "github-actions"],
       "matchPackageNames": [
         "gitea.home.keithjackson.dev/home-network/toolbox",
         "gitea.home.keithjackson.dev/home-network/agent-runner",
@@ -316,9 +318,9 @@
       "enabled": false
     },
     {
-      "description": "ADDED 2026-10-30. MIRRORS catalog/<svc>.yaml `update.range` — the operator's declared per-service policy (schemas/servicedefinition.schema.json), which Renovate does not read. Without it the dry run proposed gitea 28.2.0, qbittorrent 20.04.1 (both stale/odd tags, not real successors), radarr 6.x and nextcloud 35 (four majors past 31; Nextcloud cannot skip majors). KEEP IN STEP with the catalog ranges.",
+      "description": "ADDED 2026-10-30. MIRRORS catalog/<svc>.yaml `update.range` — the operator's declared per-service policy (schemas/servicedefinition.schema.json), which Renovate does not read. Without it the dry run proposed gitea 28.2.0, qbittorrent 20.04.1 (the qbittorrent tag is a stale/odd one, not a real successor; gitea 28.x IS the real successor of 1.27 — upstream dropped the \"1.\" prefix at 28.0.0 — and the catalog moved to it on 2026-11-07, so the gitea ceiling is now <29.0.0: 28.x patches/minors are proposed, never auto-merged per the blast-radius rule, and 29 waits for a human), radarr 6.x and nextcloud 35 (four majors past 31; Nextcloud cannot skip majors). KEEP IN STEP with the catalog ranges.",
       "matchPackageNames": ["gitea/gitea"],
-      "allowedVersions": "<2.0.0"
+      "allowedVersions": "<29.0.0"
     },
     { "matchPackageNames": ["nextcloud"], "allowedVersions": "<32.0.0" },
     { "matchPackageNames": ["lscr.io/linuxserver/qbittorrent"], "allowedVersions": "<6.0.0" },

runbooks/backup-failure.md

@@ -99,6 +99,16 @@ infra/devcontainer/dev.sh run bash -c \
      -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
 ```
 
+The `velero` CLI is in the toolbox (pinned to the server version) and gives the per-item errors and
+warnings the CR only counts. Read-only; `velero backup create/delete` and `velero restore` are
+operator actions:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'velero --kubecontext k3s-prod -n velero backup describe <name> --details; \
+   velero --kubecontext k3s-prod -n velero backup logs <name> | grep -iE "level=(error|warn)" | tail -25'
+```
+
 The plugin pin (cause 1 depends entirely on this):
 
 ```