feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs

The in-cluster runner ran at the default capacity 1 with no config file, so
gate / images / renovate runs on different refs queued behind each other
(29-759s waits; run 472 waited 759s behind a renovate burst).

- ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via
  CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault()
  applies the same defaults to a partial file as to no file; the
  generate-config example is deliberately not used (its labels would override
  the registered ones, force_pull differs from the default).
- Pod-template config-revision annotation so ConfigMap-only edits can roll it.
- dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi.
  Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs.
- runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1).

apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the
renderer only places it in a wave), so it is the source, not generated output.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.gitea/workflows/gate.yml

@@ -80,7 +80,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           fetch-depth: 0        # full history so ci/touched-requirements.sh can diff vs main
 

.gitea/workflows/images.yaml

@@ -107,7 +107,7 @@ jobs:
     timeout-minutes: 75
     steps:
       - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           fetch-depth: 0   # the per-image change detection diffs against the base commit
 

.gitea/workflows/merge-gate.yaml

@@ -60,9 +60,9 @@ jobs:
   spec-immutability:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262                 # v4                 # the implementation repo
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1                 # v7                 # the implementation repo
       - name: Check out the read-only spec (sibling path)
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           repository: ${{ vars.SPEC_REPO_SLUG }}  # e.g. Home-Network/home-network-k3s (set in Gitea)
           path: ../home-network-k3s
@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [spec-immutability]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }          # full history so touched-requirement diff works
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh lint
@@ -85,7 +85,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [lint]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l0
@@ -96,7 +96,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [l0]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -107,7 +107,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [render-golden]
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -115,7 +115,7 @@ jobs:
         run: make -C ci l1
       - name: Upload L1 artifacts
         if: always()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: l1-junit-coverage
           path: |
@@ -131,7 +131,7 @@ jobs:
     needs: [l1]
     if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh conformance
@@ -142,7 +142,7 @@ jobs:
         run: make -C ci conformance
       - name: Upload L2/L3 artifacts
         if: always()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: conformance-junit-coverage
           path: |
@@ -160,7 +160,7 @@ jobs:
     needs: [lint, l0, render-golden, l1, conformance]
     if: always()
     steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       - name: Decide
         env:
           R_LINT: ${{ needs.lint.result }}

apps/wave-1/gitea-actions.yaml

@@ -77,6 +77,42 @@ spec:
     requests:
       storage: 1Gi
 ---
+# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
+# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
+# unset field with the same default it uses when there is no file at all, so a file holding
+# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
+# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
+# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
+# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
+# no-file default (false); and its sample `envs` would be injected into every job.
+#
+# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
+# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
+# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
+# below is consumed by the image's run.sh for registration, not by that legacy path.
+#
+# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
+# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
+# config-revision annotation below) or the new value stays inert until the next restart.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: act-runner-config
+  namespace: gitea-actions
+  labels:
+    app.kubernetes.io/name: act-runner
+data:
+  config.yaml: |
+    runner:
+      # Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
+      # 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
+      # renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
+      # containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
+      # collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
+      # unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
+      # and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
+      capacity: 2
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -95,6 +131,10 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: act-runner
+      annotations:
+        # Bump when act-runner-config changes without any other pod-template change, so the
+        # Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
+        homelab.keithjackson.dev/act-runner-config-revision: "1"   # capacity 2
     spec:
       # CI holds no cluster credentials — do not mount a SA token into the runner or
       # (via the shared netns) the job containers (R-SEC-02, docs/14).
@@ -146,12 +186,23 @@ spec:
               mountPath: /certs
             - name: dind-storage
               mountPath: /var/lib/docker
+          # Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
+          # container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
+          # capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
+          #   cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
+          #     parallelised, so per-job demand rises. The request is what the CFS share is
+          #     weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
+          #     so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
+          #     requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
+          #   memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
+          #     takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
+          #     over that for the parallelised gate. Nodes have ~31Gi allocatable.
           resources:
             requests:
-              cpu: 100m
-              memory: 256Mi
+              cpu: "2"
+              memory: 1Gi
             limits:
-              memory: 2Gi
+              memory: 4Gi
           startupProbe:
             exec:
               command:
@@ -195,23 +246,36 @@ spec:
               value: /certs/client
             - name: DOCKER_TLS_VERIFY
               value: "1"
+            # The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
+            # `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
+            - name: CONFIG_FILE
+              value: /etc/act_runner/config.yaml
           volumeMounts:
             - name: data
               mountPath: /data
             - name: certs
               mountPath: /certs
               readOnly: true
+            - name: config
+              mountPath: /etc/act_runner
+              readOnly: true
+          # Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
+          # against the old 512Mi limit. It holds per-task state, log buffers and the embedded
+          # actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
           resources:
             requests:
               cpu: 100m
-              memory: 128Mi
+              memory: 256Mi
             limits:
-              memory: 512Mi
+              memory: 1Gi
       volumes:
         - name: data
           persistentVolumeClaim:
             claimName: act-runner-data
         - name: certs
           emptyDir: {}
+        - name: config
+          configMap:
+            name: act-runner-config
         - name: dind-storage
           emptyDir: {}

decisions/2026-11-07-devcontainer-velero-cli.md

@@ -0,0 +1,54 @@
+---
+date: 2026-11-07
+branch: feat/devcontainer-velero-cli
+title: Bake the velero CLI into the dev toolbox, pinned to the prod server version
+---
+
+## What changed
+
+- `infra/devcontainer/versions.env`: new `VELERO_VERSION=1.18.2`, with a comment explaining that
+  the CLI and server are version-coupled, so this pin follows the server and is not bumped on
+  its own. It has **no** `# renovate:` marker on purpose. When a `velero` chart bump in
+  `platform/velero/helmrelease.yaml` changes the appVersion, bump this pin in the same PR.
+- `infra/devcontainer/Containerfile`: `ARG VELERO_VERSION`; the binary comes from the
+  vmware-tanzu GitHub release tarball and is installed in the same RUN as flux/helm. There is
+  no checksum check, which matches the other tools in that file (none of them verify one).
+  `velero=` is recorded in `/etc/dev-toolchain-versions`. The verify stage now checks that
+  `velero` is on PATH, and `check velero v${VELERO_VERSION}` asserts the output of
+  `velero version --client-only`.
+- `infra/devcontainer/build-image.sh`: `VELERO_VERSION` added to `BUILD_ARGS`. CI reads that
+  list generically through `--print-build-args` (`.gitea/workflows/scripts/images.sh`), so
+  **no sensitive path (`.gitea/**`, `ci/**`) was touched**.
+- `infra/devcontainer/README.md`: added a Backups row to the tool table.
+- `runbooks/backup-failure.md`: under "First commands", a read-only
+  `velero backup describe --details` / `backup logs` example run through dev.sh.
+
+## What I verified
+
+- Live server version, read-only: `kubectl --context k3s-prod -n velero get deploy velero`
+  shows image `docker.io/velero/velero:v1.18.2`, and HelmRelease `velero` has
+  lastAttemptedRevision `12.2.3`.
+- Local build: `IMAGE=localhost/home-k3s-dev:velero-wt ./build-image.sh` returned rc 0. The
+  verify stage printed `ok velero` (on PATH), `ok velero v1.18.2` (pin check) and
+  `TOOLBOX VERIFIED`. The image was built under a separate local tag and **not pushed**.
+- Smoke test with that image against prod (read-only): `velero version` reports Client v1.18.2
+  and Server v1.18.2. `velero backup get` lists the daily backups, all `Completed` with 0 errors.
+- Recurring warning (reported only, not fixed): every daily backup since
+  `velero-daily-20261031030050` has 1 warning. Backups up to 20261030 have 0. The warning text is:
+  `gitea-actions: resource: /pods name: /act-runner-5944dfdfb8-69bvq message: /Skip pod volume
+  data error: /pod is not in the expected status, ... phase=Failed: pod is not running`.
+  It is the same pod in 20261031, 20261104 and 20261108. `gitea-actions` holds 11 `Failed`
+  pods with reason `Evicted` from ReplicaSet `act-runner-5944dfdfb8`, ten of them started
+  2026-10-30T08:26Z. The running runner is `act-runner-fc597f6b5-vsvkl`. So the warning comes from a
+  leftover evicted (`Failed`) pod that Velero still tries to back up volumes for. Cleaning up the evicted pods is an
+  operator action.
+- Gate: see the PR description for the toolbox `ci/gate.sh` rc and the skipped stages.
+
+## What I did NOT verify
+
+- The CI image build (`.gitea/workflows/images.yaml`) on this branch. It runs on push/PR and I
+  have not seen its result while writing this.
+- An arm64 build. Only amd64 was built (`TARGETARCH` defaults to amd64).
+- Any mutating velero command (`backup create`, `restore`, `delete`). None were run.
+- Why only `-69bvq` of the 11 evicted act-runner pods shows up in the warning. A likely guess is
+  that only that pod's volumes match the backup's pod-volume selection, but I did not check.

decisions/2026-11-08-runner-capacity-2.md

@@ -0,0 +1,155 @@
+---
+date: 2026-11-08
+branch: feat/runner-capacity-2
+title: act_runner runs two jobs at once, and dind is sized for two
+---
+
+## What changed
+
+`apps/wave-1/gitea-actions.yaml` only. That file is a **hand-owned** non-catalog unit
+(`EXTRA_UNITS["gitea-actions"]` in `catalog/render.py`): the renderer only decides which wave
+it lands in, it does not generate its bytes, and `ci/render-golden.sh` compares only files the
+catalog renders. So it is edited in place, and no re-render was needed or run.
+
+1. **New ConfigMap `act-runner-config`** holding a config file with ONE key,
+   `runner.capacity: 2`, mounted read-only at `/etc/act_runner/` and selected with
+   `CONFIG_FILE=/etc/act_runner/config.yaml`. The image's `run.sh` (act_runner v0.6.1) turns
+   that into `--config` for both `register` and `daemon`.
+   - Deltas only, on evidence: v0.6.1 `LoadDefault()` unmarshals the file onto a zero Config and
+     then applies the same defaults as the no-file path, so the other settings are unchanged.
+   - The `generate-config` example was deliberately NOT used. Its `runner.labels` would override
+     the registered labels in `/data/.runner` and repoint `ubuntu-latest` at
+     `docker.gitea.com/runner-images`. Its `force_pull: true` is not the default. Its sample `envs`
+     would be injected into every job.
+   - With a file present, act_runner ignores the legacy `GITEA_RUNNER_CAPACITY` /
+     `_FILE` / `_ENVIRON` / `_ENV_FILE` / `GITEA_DEBUG` / `GITEA_TRACE` env vars. None of
+     them is set on this Deployment.
+2. **Pod-template annotation `homelab.keithjackson.dev/act-runner-config-revision: "1"`.**
+   act_runner reads its config once, at start, so a change to the ConfigMap alone would never
+   take effect. Bump this annotation whenever you change only the ConfigMap.
+3. **dind resources:** requests `cpu 100m -> 2` and `memory 256Mi -> 1Gi`; limit
+   `memory 2Gi -> 4Gi`. There is still no CPU limit, on purpose.
+4. **runner resources:** request `memory 128Mi -> 256Mi`, limit `memory 512Mi -> 1Gi`. CPU stays
+   at `100m`.
+
+Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1:
+
+| container | peak CPU (1m rate, 30d) | peak working set (30d) | OOM events (7d) |
+|---|---|---|---|
+| dind | 0.69 core | 447 MiB | 0 |
+| runner | 0.24 core | 314 MiB | 0 |
+
+At the moment of measurement the pod was idle (`top`: dind 1m/138Mi, runner 1m/83Mi), on
+k3s-prod-2.
+
+- **CPU request 2.** Two jobs at today's peak is about 1.4 cores. The gate is being parallelised
+  in other work, so each job will use more than that. The request sets dind's CFS weight under
+  contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an
+  idle node.
+- **Memory 1Gi / 4Gi.** Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed,
+  dockerd goes down and takes both running jobs with it, so the limit is about 4x that.
+- **Runner limit 512Mi -> 1Gi.** Its 30-day peak was already 61% of the old limit, and with
+  capacity 2 it holds state for two tasks.
+
+Scheduling headroom: CPU requests per node were measured with `describe node`. The pod's total
+CPU request goes from 200m to 2100m (+1.9 cores).
+
+| node | CPU requested now | after the change | allocatable |
+|---|---|---|---|
+| prod-2 (current) | 2945m | ~4845m (61%) | 8 |
+| prod-1 | 3175m | ~5075m | 8 |
+| prod-3 | 3840m | ~5740m (72%) | 8 |
+
+The pod fits on any of the three if the RWO PVC moves it.
+
+Concurrency semantics. Workflow `concurrency:` groups are enforced by Gitea on the server, not by
+the runner, so capacity does not change them:
+
+- `gate.yml` `gate-${{ github.ref }}` with cancel-in-progress still cancels a superseded run on
+  the same ref.
+- `renovate` (group `renovate`, no cancel) and `images-${{ github.ref }}` still serialise
+  themselves.
+
+What capacity 2 does change: runs on *different* refs (a PR gate next to a renovate or images
+run, or two PRs) now run side by side instead of queueing. The required check
+`gate / gate (pull_request)` is still one job per PR head, and gate.yml's other job
+(`conformance`) is `if: false`, so it never runs.
+
+Both jobs share one dockerd. act_runner gives each task its own container prefix
+(`GITEA-ACTIONS-TASK-<task id>`) and its own network, and no workflow uses fixed container names
+or host ports.
+
+## What I verified
+
+- **Source of truth.** `catalog/render.py` `EXTRA_UNITS` places `gitea-actions.yaml` by
+  anchor/offset only. `ci/render-golden.sh` iterates over rendered files only, so this file is
+  outside render==golden. No `--emit-waves` was run.
+- **act_runner v0.6.1 behaviour, read in its source** (tag v0.6.1):
+  - `config.LoadDefault()` treats a partial file as described above.
+  - `daemon.go`: config labels override `.runner` labels only when non-empty.
+  - `poller.go`: a semaphore of `cfg.Runner.Capacity` bounds concurrent tasks.
+  - `runner.go`: per-task container name prefix.
+  - The image's `run.sh`, read from `gitea/act_runner:0.6.1`, honours `CONFIG_FILE`.
+- **The live pod has no config file today.** `/data` holds only `.runner` and `lost+found`.
+  `act_runner --version` reports v0.6.1. No LimitRange or ResourceQuota exists in
+  `gitea-actions`.
+- **Resource numbers.** All figures above come from read-only `kubectl top`, `describe node` and
+  Prometheus queries against k3s-prod.
+- **dind disk.** The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G
+  free. Image cache is 2.18GB, which is the one `catthehacker/ubuntu:act-22.04` image. Two
+  concurrent jobs share that image.
+- **Embedded config.** It parses as YAML to `{'runner': {'capacity': 2}}`. The only `$` in the
+  manifest is the existing `${BASE_DOMAIN}`, so nothing new is exposed to Flux postBuild
+  envsubst.
+- **Approval policy.** `apps/wave-1/` matches no `sensitive_paths` glob in
+  `infra/gitea/approval-policy.json`, so this PR merges on a green gate alone.
+- **Local gate.** Pinned toolbox, run from the worktree on the committed tree. The rc and skipped
+  stages are given in the PR description.
+
+## What I did NOT verify
+
+- **UNVERIFIED: that the runner actually starts with the config and runs two jobs at once.**
+  Nothing is applied until Flux reconciles `main`.
+- **Capacity cannot be seen through Gitea's API.** act_runner v0.6.1 does not report capacity to
+  Gitea: `Declare` sends only version and labels. `GET /api/v1/admin/actions/runners` therefore
+  cannot show "capacity 2". Concurrency has to be observed instead.
+- **UNVERIFIED: whether the parallelised gate fits in these sizes.** That gate has not landed.
+  Re-measure after it does, with
+  `max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s])`.
+- **Rollout cancels in-flight jobs.** The rollout restarts the runner pod (strategy Recreate,
+  `shutdown_timeout: 0s`), and any job running at that moment is cancelled, possibly including
+  the `push`-to-main gate run triggered by this very merge. Re-run it if that happens.
+- **dind disk is not bounded by a limit.** With two jobs it holds two jobs' workspaces and layers
+  on the same 11.8G node filesystem. I did not add an `ephemeral-storage` limit, because an
+  eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node.
+- **Stale comments in sensitive paths.** `.gitea/workflows/gate.yml` (line ~52) and
+  `.gitea/workflows/README.md` (line ~86) still say "`act_runner` runs at capacity 1". I left
+  them alone because `.gitea/**` is a sensitive path and editing them would put this PR behind
+  human approval. Fix them in a separate PR.
+
+### Operator post-merge checklist
+
+1. Flux applied it:
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}'`
+   equals the merge commit.
+2. The pod restarted with the config:
+   - `kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner`
+     shows a new pod, 2/2 Running.
+   - `kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml` shows
+     `capacity: 2`.
+   - `kubectl ... exec deploy/act-runner -c runner -- ps` shows `act_runner daemon --config /etc/act_runner/config.yaml`.
+   - The logs show "declare successfully" with the two expected labels (`ubuntu-latest`,
+     `homelab` -> `catthehacker/ubuntu:act-22.04`). That confirms the labels did not change.
+3. The runner is registered and online:
+   - `GET /api/v1/admin/actions/runners` lists `incluster-dind`, online, with the same ID as
+     before (no duplicate row).
+   - Capacity is NOT shown there; see above.
+4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a
+   `workflow_dispatch` of renovate). Then either:
+   - in `GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks`, confirm their
+     `run_started_at` .. `updated_at` windows overlap; or
+   - check
+     `docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps` in the dind container lists two
+     `GITEA-ACTIONS-TASK-*` container sets at once.
+5. Resources: `kubectl --context k3s-prod -n gitea-actions describe pod` shows dind requests
+   cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.

infra/devcontainer/Containerfile

@@ -30,6 +30,9 @@ ARG CHECK_JSONSCHEMA_VERSION
 # Pinned to the Prometheus version prod actually runs — see versions.env for why that lockstep is
 # the point, and for the live-cluster command that establishes it.
 ARG PROMTOOL_VERSION
+# velero: operator CLI for the backup server. Pinned to the server version prod runs — the CLI
+# and server are version-coupled; see versions.env.
+ARG VELERO_VERSION
 ARG TARGETARCH=amd64
 
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
@@ -66,6 +69,10 @@ RUN set -eux; \
     curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" -o /usr/local/bin/kubectl; chmod +x /usr/local/bin/kubectl; \
     curl -fsSL "https://get.helm.sh/helm-v${HELM_VERSION}-linux-${TARGETARCH}.tar.gz" | tar -xz -C /tmp; \
       install -m0755 "/tmp/linux-${TARGETARCH}/helm" /usr/local/bin/helm; \
+    VELERO_TARBALL="velero-v${VELERO_VERSION}-linux-${TARGETARCH}"; \
+    curl -fsSL "https://github.com/vmware-tanzu/velero/releases/download/v${VELERO_VERSION}/${VELERO_TARBALL}.tar.gz" \
+      | tar -xz -C /usr/local/bin --strip-components=1 "${VELERO_TARBALL}/velero"; \
+      chmod +x /usr/local/bin/velero; \
     rm -rf /tmp/*
 
 # Pinned conformance test toolchain (L0/L1 static gate + the L2/L3 Chainsaw runner).
@@ -93,7 +100,7 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
       echo "kyverno=${KYVERNO_VERSION}"; echo "kyverno-json=${KYVERNO_JSON_VERSION}"; \
       echo "chainsaw=${CHAINSAW_VERSION}"; echo "yamllint=${YAMLLINT_VERSION}"; \
       echo "yq=${YQ_VERSION}"; echo "check-jsonschema=${CHECK_JSONSCHEMA_VERSION}"; \
-      echo "promtool=${PROMTOOL_VERSION}"; \
+      echo "promtool=${PROMTOOL_VERSION}"; echo "velero=${VELERO_VERSION}"; \
     } > /etc/dev-toolchain-versions
 
 WORKDIR /work
@@ -112,6 +119,7 @@ ARG KUBECTL_VERSION
 ARG CHAINSAW_VERSION
 ARG KYVERNO_VERSION
 ARG SOPS_VERSION
+ARG VELERO_VERSION
 RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
 echo "--- no network (the build-time stand-in for run --network none) ---"
 if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
@@ -122,13 +130,13 @@ echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
 echo "--- every tool is on PATH ---"
 for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
          yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
-         wg qrencode; do
+         wg qrencode velero; do
   if command -v "$t" >/dev/null 2>&1; then printf "    ok      %s\n" "$t";
   else printf "    MISSING %s\n" "$t"; missing=1; fi
 done
 [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
 echo "--- version-sensitive tools report their pins ---"
-: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
+: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}" "${VELERO_VERSION:?}"
 check() { # name pin actual-output
   if printf "%s" "$3" | grep -qF "$2"; then printf "    ok      %-10s %s\n" "$1" "$2";
   else printf "    DRIFT   %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
@@ -138,6 +146,7 @@ check kubectl  "${KUBECTL_VERSION}"  "$(kubectl version --client 2>&1)"
 check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
 check kyverno  "${KYVERNO_VERSION}"  "$(kyverno version 2>&1)"
 check sops     "${SOPS_VERSION}"     "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
+check velero   "v${VELERO_VERSION}"  "$(velero version --client-only 2>&1)"
 [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
 echo "--- python can import what the renderer and the gate need ---"
 python3 -c "import yaml; print('    PyYAML', yaml.__version__)"

infra/devcontainer/README.md

@@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses.
 | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
 | GitOps / cluster | `flux`, `kubectl`, `helm` |
 | Secrets | `sops`, `age` + `age-keygen` |
+| Backups (`platform/velero`) | `velero` CLI — pinned to the **server** version prod runs (version-coupled; see `versions.env`) |
 | VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
 | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
 | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |

infra/devcontainer/build-image.sh

@@ -54,6 +54,7 @@ BUILD_ARGS=(
   "KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
   "YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
   "CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
+  "VELERO_VERSION=$VELERO_VERSION"
 )
 if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
 

infra/devcontainer/versions.env

@@ -44,4 +44,19 @@ CHECK_JSONSCHEMA_VERSION=0.29.4
 # promtool ships INSIDE the prometheus release tarball (prometheus-<v>.linux-<arch>.tar.gz), so
 # the version string here is a Prometheus release, not a separate promtool release.
 PROMTOOL_VERSION=3.14.0
+# velero — the operator CLI for the Velero backup server (platform/velero): `velero backup
+# describe <name> --details`, `velero backup logs`, `velero backup create --from-schedule`.
+#
+# THIS PIN IS NOT FREE-FLOATING: the CLI and server are VERSION-COUPLED (the CLI talks to the
+# server's CRDs and its download-request protocol; upstream supports only a matching minor), so
+# it MUST equal the Velero server version actually running on prod.
+# Verified against the live cluster on 2026-11-07:
+#   kubectl --context k3s-prod -n velero get deploy velero \
+#     -o jsonpath='{.spec.template.spec.containers[0].image}'
+#     -> docker.io/velero/velero:v1.18.2   (HelmRelease velero, chart velero-12.2.3)
+# Deliberately carries NO `# renovate:` marker: it must move WITH the server, i.e. when a
+# vmware-tanzu `velero` chart bump (platform/velero/helmrelease.yaml, range >=12 <13) changes the
+# chart's appVersion, bump this in the same PR. Never bump it on its own. The verify stage
+# asserts `velero version --client-only` reports exactly this version.
+VELERO_VERSION=1.18.2
 # Proxmox provider (bpg/proxmox) is pinned in the tofu module (infra/tofu/versions.tf), not here.

platform/keycloak/kustomization.yaml

@@ -28,11 +28,11 @@ resources:
   # branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
   # passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
-  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
-  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
   # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
-  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/kubernetes.yml
+  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml
   # --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
   - keycloak-admin-credentials.sops.yaml
   # --- OIDC client secrets (grafana + oauth2-proxy) the realm import injects via placeholders ---

runbooks/backup-failure.md

@@ -99,6 +99,16 @@ infra/devcontainer/dev.sh run bash -c \
      -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
 ```
 
+The `velero` CLI is in the toolbox (pinned to the server version) and gives the per-item errors and
+warnings the CR only counts. Read-only; `velero backup create/delete` and `velero restore` are
+operator actions:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'velero --kubecontext k3s-prod -n velero backup describe <name> --details; \
+   velero --kubecontext k3s-prod -n velero backup logs <name> | grep -iE "level=(error|warn)" | tail -25'
+```
+
 The plugin pin (cause 1 depends entirely on this):
 
 ```