Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5.5
|
ea30ba876c |
No files matched your search
@@ -165,8 +165,7 @@ Cost: the job shares the single act_runner slot with the gate, but the build its
|
||||
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
|
||||
images.yaml build/verify/publish this repo's container images (see above)
|
||||
scripts/
|
||||
install-tools.sh installs the pinned tools per stage (from ci/versions.env) — used by the
|
||||
dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
|
||||
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
|
||||
decide.sh the R-TEST-03 merge decision over job results
|
||||
images.sh the image pipeline's driver (what to build, base, push, digest)
|
||||
README.md this file
|
||||
|
||||
+10
-99
@@ -3,10 +3,8 @@
|
||||
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
|
||||
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
|
||||
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
|
||||
# (R-TEST-02) — and since 2026-11-08 it runs INSIDE the same image: the job `container:` is the
|
||||
# CI-built devcontainer toolbox (see "THE JOB IMAGE" below), not a generic runner image that
|
||||
# downloads the pinned tools on every run. (See further below for why the script, not `make`,
|
||||
# is invoked here.) It runs every stage in order —
|
||||
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
|
||||
# below for why the script, not `make`, is invoked here). It runs every stage in order —
|
||||
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
|
||||
# R-TEST-03 decision, printing
|
||||
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
|
||||
@@ -19,8 +17,7 @@
|
||||
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
|
||||
# environment it is:
|
||||
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
|
||||
# baked into the job image and checked against ci/versions.env two steps up.
|
||||
# GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
|
||||
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
|
||||
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
|
||||
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
|
||||
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
|
||||
@@ -44,32 +41,6 @@
|
||||
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
|
||||
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
|
||||
# is retained dormant in merge-gate.yaml.
|
||||
#
|
||||
# THE JOB IMAGE (2026-11-08, feat/gate-toolbox-container). The gate job runs in
|
||||
# gitea.home.keithjackson.dev/home-network/toolbox, pinned BY DIGEST — the image
|
||||
# .gitea/workflows/images.yaml builds from infra/devcontainer/Containerfile, verifies, and
|
||||
# publishes. It is the SAME image `infra/devcontainer/dev.sh` runs, so a local gate and the CI
|
||||
# gate now execute the identical toolchain (R-TEST-02), not two installs that agree by intent.
|
||||
# Before this, the job ran in ghcr.io/catthehacker/ubuntu:act-22.04 and spent ~11s of every run
|
||||
# re-downloading yamllint, check-jsonschema, kyverno, promtool and chainsaw
|
||||
# (.gitea/workflows/scripts/install-tools.sh l0 + l1; measured on runs 494/495).
|
||||
# * PINNED BY DIGEST, MOVED BY RENOVATE. The `# renovate-digest:` marker above the image line
|
||||
# is the repo-wide convention (renovate.json5, "This repo's own CI-published images"):
|
||||
# Renovate follows the `main` tag and opens a digest-bump PR, whose OWN gate run then
|
||||
# executes on the new image — the bump tests itself. infra/gitea/registry-retention.sh reads
|
||||
# the same marker, so the pinned digest is never pruned. Never hand-edit it to a tag.
|
||||
# * PULL AUTH: none needed. The Home-Network org is public and the registry issues an
|
||||
# anonymous pull token (measured 2026-11-08: anonymous /v2/token -> manifest 200), so the
|
||||
# runner's DinD pulls it like any public image. If the org is ever made private, add
|
||||
# `credentials:` here with a read:package token secret — the automatic Actions token has no
|
||||
# package scope at /v2/token (see images.yaml's header).
|
||||
# * COLD PULL: ~467 MB compressed (1.54 GB unpacked). The DinD store is an emptyDir, so the
|
||||
# first job after a runner-pod restart or a digest bump pays one pull; every other run finds
|
||||
# it cached (forcePull=false), exactly as it already did for the catthehacker image.
|
||||
# * NO NODE in the toolbox, so `actions/checkout` (a Node action) cannot run here. The checkout
|
||||
# step is plain git instead — see its comment for how it keeps fetch-depth-0 semantics.
|
||||
# * install-tools.sh is NOT deleted: merge-gate.yaml (the dormant blueprint) still calls it,
|
||||
# and conformance/tests/l0-static/alert-rules-unit-tested asserts its promtool wiring.
|
||||
---
|
||||
name: gate
|
||||
|
||||
@@ -107,76 +78,16 @@ jobs:
|
||||
# Single required status check: the static merge gate.
|
||||
gate:
|
||||
runs-on: ubuntu-latest
|
||||
# The CI-built toolbox — the same image dev.sh runs. See "THE JOB IMAGE" in the header.
|
||||
container:
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
|
||||
image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
|
||||
steps:
|
||||
# Plain git, because actions/checkout is a Node action and the toolbox has no Node. It
|
||||
# reproduces what actions/checkout@v7 with `fetch-depth: 0` did here (read from its logged
|
||||
# commands on runs 490/495): full history of every branch + tags, so
|
||||
# ci/touched-requirements.sh can diff against origin/<base>; the exact event commit
|
||||
# checked out (detached for a PR, as `-B <branch>` for a push). One difference, on
|
||||
# purpose: the job token is passed to the single fetch through GIT_CONFIG_* environment
|
||||
# variables and is NOT persisted into .git/config — nothing after this step needs
|
||||
# authenticated git (ci/ and conformance/ make no network git calls), so there is no
|
||||
# reason to leave a credential in the workspace for the gate's scripts to read.
|
||||
- name: Checkout (git, full history)
|
||||
env:
|
||||
GIT_TOKEN: ${{ github.token }}
|
||||
SERVER_URL: ${{ github.server_url }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
SHA: ${{ github.sha }}
|
||||
REF: ${{ github.ref }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# act runs steps with umask 0000, so without this every checked-out directory is
|
||||
# world-writable — and Ansible then IGNORES infra/ansible/ansible.cfg as a source
|
||||
# ("Ansible is being run in a world writable directory"), which made the L0 check
|
||||
# prod-baremetal-not-dev-surface fail on this PR's first CI run (run 500). The
|
||||
# toolbox has Ansible, so that check now drives it for real. 0022 = what a
|
||||
# workstation clone gets (reproduced locally: 0000 fails the check, 0022 passes).
|
||||
umask 0022
|
||||
git config --global --add safe.directory "$PWD"
|
||||
git init -q .
|
||||
git remote add origin "${SERVER_URL}/${REPOSITORY}"
|
||||
auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
|
||||
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
|
||||
GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
|
||||
git -c protocol.version=2 fetch --quiet --no-tags --prune --no-recurse-submodules origin \
|
||||
'+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' \
|
||||
"+${SHA}:refs/remotes/checkout/head"
|
||||
unset auth GIT_TOKEN
|
||||
case "$REF" in
|
||||
refs/heads/*) git checkout -q --force -B "${REF#refs/heads/}" "$SHA" ;;
|
||||
*) git checkout -q --force --detach "$SHA" ;;
|
||||
esac
|
||||
[ "$(git rev-parse HEAD)" = "$SHA" ] || { echo "::error::HEAD is not ${SHA}" >&2; exit 1; }
|
||||
echo "checked out $(git log -1 --format='%H %s') (ref ${REF}); $(git rev-list --count HEAD) commits of history"
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
fetch-depth: 0 # full history so ci/touched-requirements.sh can diff vs main
|
||||
|
||||
# The image is the toolchain, so prove it is the PINNED one before trusting it: every
|
||||
# *_VERSION in ci/versions.env must equal what the image recorded at build time
|
||||
# (/etc/dev-toolchain-versions, written by infra/devcontainer/Containerfile; its `verify`
|
||||
# stage already asserted the version-sensitive binaries report those pins). This replaces
|
||||
# the per-run install: a drifted toolbox fails HERE, naming the tool, instead of gating with
|
||||
# a different engine. ci/versions.env and infra/devcontainer/versions.env must move together.
|
||||
- name: Toolchain is the one ci/versions.env pins
|
||||
- name: Install pinned CI toolchain (ci/versions.env)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
. ci/versions.env
|
||||
drift=0
|
||||
for var in $(grep -oE '^[A-Z_]+_VERSION' ci/versions.env); do
|
||||
tool="$(printf '%s' "${var%_VERSION}" | tr 'A-Z_' 'a-z-')"
|
||||
want="${!var}"
|
||||
got="$(sed -n "s/^${tool}=//p" /etc/dev-toolchain-versions)"
|
||||
if [ "$got" = "$want" ]; then
|
||||
printf 'ok %-18s %s\n' "$tool" "$want"
|
||||
else
|
||||
printf '::error::DRIFT %s: ci/versions.env pins %s, the job image has %s\n' "$tool" "$want" "${got:-nothing}"
|
||||
drift=1
|
||||
fi
|
||||
done
|
||||
[ "$drift" -eq 0 ]
|
||||
./.gitea/workflows/scripts/install-tools.sh l0
|
||||
./.gitea/workflows/scripts/install-tools.sh l1
|
||||
|
||||
# R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
|
||||
# spec-immutability check, so its result is a distinct, tolerant status line in the
|
||||
|
||||
+1
-1
@@ -99,7 +99,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|
||||
|
||||
| stage | a skip there means | verdict |
|
||||
|---|---|---|
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is the job image — the CI-built toolbox, `.gitea/workflows/gate.yml` `container:` — so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
||||
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
|
||||
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
||||
|
||||
+3
-3
@@ -46,8 +46,8 @@
|
||||
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
|
||||
#
|
||||
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
|
||||
# seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
|
||||
# CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
|
||||
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
|
||||
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
|
||||
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
|
||||
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
|
||||
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
|
||||
@@ -257,7 +257,7 @@ for s in lint l0 render-golden l1; do
|
||||
skip)
|
||||
if strict_skips; then
|
||||
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
|
||||
err " toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
|
||||
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
|
||||
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
|
||||
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
|
||||
BLOCK=1
|
||||
|
||||
@@ -97,9 +97,9 @@ cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
|
||||
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
|
||||
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
|
||||
#
|
||||
# CI the pinned toolchain IS the job image (the CI-built toolbox; gate.yml `container:`),
|
||||
# and every seam is on `main`. A stage that skips here is a
|
||||
# DEFECT (a tool dropped from the image, a seam that went unwired) -> BLOCK.
|
||||
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
|
||||
# before the gate runs, and every seam is on `main`. A stage that skips here is a
|
||||
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
|
||||
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
|
||||
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
|
||||
#
|
||||
|
||||
@@ -1,104 +0,0 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: feat/gate-toolbox-container
|
||||
title: The merge gate runs inside the CI-built toolbox image, pinned by digest — no per-run tool install
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`.gitea/workflows/gate.yml`, job `gate` (job name, the gate step, its env and its
|
||||
`|| [ "$?" -eq 3 ]` tolerance are unchanged, so the required context `gate / gate (pull_request)`
|
||||
is unchanged):
|
||||
|
||||
- **`container:`** `gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e…ef81` — the
|
||||
image `images.yaml` published from `dd94101` (run 789; it is today's `main` and `sha-dd941016478c`
|
||||
tag). The same image `infra/devcontainer/dev.sh` runs, so local and CI gate on the identical
|
||||
toolchain. A `# renovate-digest: image=… tag=main` marker sits on the line above, the repo's
|
||||
existing convention; `infra/gitea/registry-retention.sh` reads the same marker, so this digest
|
||||
is protected from pruning.
|
||||
- **`actions/checkout` replaced by a plain-git step.** The toolbox has no Node (measured:
|
||||
`command -v node` → missing), and `actions/checkout` is a Node action. Chosen over adding Node to
|
||||
the toolbox because that would need two sequential PRs (Containerfile, then a publish, then the
|
||||
pin) and would put a JS runtime into the operator toolbox only to run one action. The step
|
||||
reproduces what checkout@v7 `fetch-depth: 0` actually ran here (read from runs 490 and 495):
|
||||
every branch to `refs/remotes/origin/*`, tags, plus the event SHA; detached for a PR,
|
||||
`-B <branch>` for a push; asserts `HEAD == github.sha`. The job token goes to that one fetch via
|
||||
`GIT_CONFIG_*` env and is **not** persisted in `.git/config` (nothing later needs network git).
|
||||
It sets **`umask 0022`** first: the PR's first CI run (run 500) failed L0
|
||||
`prod-baremetal-not-dev-surface` because act runs steps with umask 0000, every checked-out
|
||||
directory was world-writable, and Ansible (present in the toolbox, absent from the catthehacker
|
||||
image) then ignored `infra/ansible/ansible.cfg`. Reproduced locally in the pinned image: clone
|
||||
under umask 0000 → that check fails with the same message; under 0022 → `check OK`. Side
|
||||
effect worth knowing: the Ansible-driven parts of that check now actually execute in CI.
|
||||
- **"Install pinned CI toolchain" step removed**, replaced by **"Toolchain is the one
|
||||
ci/versions.env pins"**: every `*_VERSION` in `ci/versions.env` must equal the image's
|
||||
`/etc/dev-toolchain-versions`, else `::error::DRIFT` and the job fails. Without this, dropping
|
||||
the installer would have made `ci/versions.env` an unenforced claim for everything except
|
||||
promtool (which `alert-rules-unit-tested` already asserts).
|
||||
- Header comments rewritten for the new job image (pull auth, cold-pull size, no-Node, why
|
||||
`install-tools.sh` stays).
|
||||
|
||||
`renovate.json5`: the `renovate-digest` custom manager now also scans `.gitea/workflows/gate.yml`;
|
||||
the `github-actions` manager is disabled for this repo's own images (same reason the `kubernetes`
|
||||
manager already was: it would see a digest-only image and look up `latest`).
|
||||
|
||||
Comment/message wording only: `ci/gate.sh` (header + the strict-skip error text), `ci/lib.sh`,
|
||||
`ci/README.md`, `.gitea/workflows/README.md` — they said the CI toolchain comes from
|
||||
`install-tools.sh`.
|
||||
|
||||
`.gitea/workflows/scripts/install-tools.sh` is **kept**: `merge-gate.yaml` (dormant blueprint)
|
||||
calls it, and `conformance/tests/l0-static/alert-rules-unit-tested/check.sh` asserts its promtool
|
||||
wiring (not touched — another agent owns `conformance/`).
|
||||
|
||||
**Consequence to know:** every toolbox digest bump is now a Renovate PR touching `.gitea/**`,
|
||||
i.e. a sensitive path needing an operator `/approve-override`. That is arguably correct (a new
|
||||
toolchain changes what green means — and `ci/versions.env` was already sensitive), but it is a
|
||||
new recurring operator touch.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Registry pull auth: the Home-Network org is `public`; anonymous `/v2/token` → manifest `200`,
|
||||
`docker-content-digest: sha256:83a3501e…ef81` for both `main` and `sha-dd941016478c`. So no
|
||||
`container: credentials:` is needed. (The Actions token could not have been used anyway — no
|
||||
package scope, per images.yaml's header.)
|
||||
- Image: 466,912,274 bytes compressed across 6 layers (manifest sum), 1,541,728,125 bytes
|
||||
unpacked (`podman image inspect`). Local cold pull 10.4s on the workstation.
|
||||
- Image tool versions equal `ci/versions.env` for all 8 pins (kyverno 1.19.0, kyverno-json 0.0.3,
|
||||
chainsaw 0.2.12, yamllint 1.35.1, kubectl 1.36.4, yq 4.44.6, promtool 3.14.0, check-jsonschema
|
||||
0.29.4) — both via `/etc/dev-toolchain-versions` and by running each binary's `--version`.
|
||||
`sleep`, `tail`, `git` 2.56.0, `bash`, `python3`, `jq`, `curl` present (act's entrypoint is
|
||||
`/bin/sleep 10800`).
|
||||
- The new checkout script, run in the pinned image against the public repo (auth header disabled
|
||||
for the test only): PR ref `refs/pull/143/head` → detached at `60e3a24`, 1018 commits; push ref
|
||||
`refs/heads/main` → `On branch main` at `e8d2666`; `origin/main` present in both. The
|
||||
toolchain-drift step passes on the real `ci/versions.env` and exits 1 naming both tools when
|
||||
yamllint/promtool pins are altered.
|
||||
- Renovate marker regex (copied from renovate.json5) matches gate.yml exactly once, capturing
|
||||
depName/tag/digest.
|
||||
- LOCAL GATE, run **in the pinned CI image itself** (`IMAGE=<the digest above>` through
|
||||
`dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding the main checkout's `.git` read-only),
|
||||
on commit `9537ca2`: **rc 3**; spec-immutability, decisions-log (78 entries), lint, l0,
|
||||
render-golden, l1, approval-policy (N/A locally — not a PR event) all pass; **gitea-settings
|
||||
the only stage not run** (no `SETTINGS_API_TOKEN` locally); L2/L3 conformance skipped as
|
||||
usual. rc 3 is not green.
|
||||
|
||||
- **CI, this PR's own runs** (Gitea API `/actions/runs/{id}/jobs` + job logs):
|
||||
- run 500 (`514ed5d`, cold DinD cache: `Image exists? false`, image pulled): job 151s; set-up
|
||||
incl. pull **9s**, checkout <1s, toolchain check 1s (all 8 pins `ok`), gate 140s → BLOCKED by
|
||||
L0 `prod-baremetal-not-dev-surface` (umask, fixed above) + approval-policy.
|
||||
- run 503 (`27d603f`, warm: `Image exists? true`): job **140s**; set-up 1s, checkout 1s,
|
||||
toolchain check 0s, gate 137s. Every stage passes — l0 (incl. prod-baremetal-not-dev-surface),
|
||||
lint, decisions-log, render-golden, l1, **gitea-settings** — except **approval-policy**,
|
||||
the expected sensitive-path block. spec-immutability skipped (expected, no spec repo in CI).
|
||||
- baseline before (runs 494/495, push on main): job 180s/179s; set-up 1–2s, checkout 1s,
|
||||
**install 11s**, gate 164–165s. The install step is gone; the remaining ~25s gate-step delta
|
||||
is NOT claimed as caused by this change (different base commits, one sample each).
|
||||
- Re-gated locally on `27d603f` in the pinned image: rc 3, gitea-settings the only not-run stage.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- Renovate actually opening a digest-bump PR for gate.yml — only the regex was tested, not a
|
||||
Renovate run.
|
||||
- Push-to-main behaviour in CI (only the PR event runs before merge).
|
||||
- Whether the gate-step speedup (165s → 137s) is real or run-to-run variance.
|
||||
- That the cold-pull cost stays ~9s under a busier node or a bigger digest delta; one sample.
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/keycloak-operator-crds
|
||||
title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
|
||||
`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
|
||||
CRDs, not two (GitHub contents API, `?ref=26.8.2`):
|
||||
|
||||
| file | at 26.0.6 | at 26.8.2 |
|
||||
|---|---|---|
|
||||
| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
|
||||
| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
|
||||
| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
|
||||
| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
|
||||
| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
|
||||
|
||||
(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
|
||||
the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
|
||||
on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
|
||||
... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
|
||||
`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
|
||||
`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
|
||||
stayed up.
|
||||
|
||||
**Fix.**
|
||||
- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
|
||||
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
|
||||
custom.regex manager matches all five — checked by running its regex over the file). Header
|
||||
comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
|
||||
- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
|
||||
450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
|
||||
so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
|
||||
restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
|
||||
`kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
|
||||
3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
|
||||
installing the CRDs is upstream's supported shape.
|
||||
- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
|
||||
"the release URLs". `renovate.json5`: same wording fix in a comment only.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
|
||||
two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
|
||||
still 450Mi/450Mi.
|
||||
- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
|
||||
`keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
|
||||
25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
|
||||
operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
|
||||
checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
|
||||
`keycloak`.
|
||||
- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
|
||||
- Gate: see the PR body for the exact rc and stages of the run against the final commit.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
|
||||
observed failing; another start-up failure behind them is possible.
|
||||
- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
|
||||
a week.
|
||||
- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
|
||||
keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
|
||||
`RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
|
||||
one. Back up / confirm the latest CNPG backup first if that matters.
|
||||
- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
|
||||
create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
|
||||
`from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
|
||||
and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
|
||||
- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
|
||||
Keycloak" list) — not re-checked here; PR #121 merged without it.
|
||||
- `catalog/keycloak.yaml`'s documentation-only image tag was not updated.
|
||||
@@ -367,9 +367,17 @@ is involved — so step 2 failing with no log entry is the expected signature of
|
||||
The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's
|
||||
`RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator
|
||||
`26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are
|
||||
released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in
|
||||
released in lockstep from `keycloak/keycloak-k8s-resources`, and the release URLs in
|
||||
`kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group,
|
||||
labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick.
|
||||
|
||||
**Renovate bumps the URLs that exist; it cannot add a CRD a release introduced.** Before merging a
|
||||
bump, diff `kubernetes/` (and its `kustomization.yml`) at the new tag against the URL list. 26.0.6
|
||||
shipped 2 CRDs, 26.8.2 ships 4 (`keycloakoidcclients`, `keycloaksamlclients` added); PR #121 bumped
|
||||
only the existing URLs and the 26.8.2 operator CrashLooped on the missing `keycloaksamlclients`
|
||||
informer, stalling Flux `platform` and every tier below it (2026-11-08,
|
||||
`decisions/2026-11-08-keycloak-operator-crds.md`). The operator's memory is also patched in
|
||||
`kustomization.yaml` (upstream's 450Mi OOMKilled 26.0.6 here); keep the patch across bumps.
|
||||
`catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and
|
||||
Renovate is disabled for it — update it by hand to match after an operator bump merges.
|
||||
|
||||
|
||||
@@ -20,18 +20,32 @@ kind: Kustomization
|
||||
namespace: keycloak
|
||||
resources:
|
||||
- namespace.yaml
|
||||
# --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
|
||||
# --- Keycloak Operator v26.8.2 (pinned upstream release manifests) ---
|
||||
# The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no
|
||||
# spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:<same tag>)
|
||||
# is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate
|
||||
# marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE
|
||||
# is what runs. These URLs are therefore THE Keycloak version. Each carries a Renovate
|
||||
# marker (custom.regex manager, renovate.json5) under ONE depName, so all of them bump in ONE
|
||||
# branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
|
||||
# passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
|
||||
#
|
||||
# THE URL LIST MUST MATCH UPSTREAM'S kubernetes/kustomization.yml AT THE NEW TAG — Renovate only
|
||||
# rewrites the version in the URLs that are here; it cannot add a CRD the new release introduced.
|
||||
# 26.0.6 shipped 2 CRDs; 26.8.2 ships 4 (+ keycloakoidcclients, keycloaksamlclients, both
|
||||
# v2alpha1). PR #121 bumped only the existing three URLs, and the 26.8.2 operator — which starts an
|
||||
# informer for EVERY kind it controls, used or not — CrashLooped on
|
||||
# `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 ... Not Found`,
|
||||
# stalling Flux `platform` and everything downstream (2026-11-08). On every bump, diff
|
||||
# https://github.com/keycloak/keycloak-k8s-resources/tree/<tag>/kubernetes
|
||||
# against this list before merging.
|
||||
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
|
||||
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
|
||||
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
|
||||
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
|
||||
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
|
||||
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakoidcclients.k8s.keycloak.org-v1.yml
|
||||
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
|
||||
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaksamlclients.k8s.keycloak.org-v1.yml
|
||||
# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
|
||||
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml
|
||||
# --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
|
||||
- keycloak-admin-credentials.sops.yaml
|
||||
@@ -51,3 +65,23 @@ resources:
|
||||
- keycloak-db-backup-credentials-prod.sops.yaml
|
||||
# --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) ---
|
||||
- grafana-dashboard.yaml
|
||||
# --- Operator memory: upstream ships requests=limits=450Mi, which this cluster outgrew ON 26.0.6 ---
|
||||
# Measured on prod 2026-11-08: the 26.0.6 operator pod had 154 restarts in 64d, lastState
|
||||
# `OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and sat at 421Mi of its 450Mi (94%) in `kubectl top`.
|
||||
# 26.8.2 keeps the same 450Mi default but runs FOUR controllers/informers instead of two (adds
|
||||
# keycloakoidcclient + keycloaksamlclient) on a newer Quarkus (3.15.1 -> 3.40.1.1), so it would
|
||||
# plausibly start above a ceiling 26.0.6 already crossed. Limit 768Mi = ~1.8x the observed 421Mi
|
||||
# working set; request 512Mi = the observed working set plus headroom, so the scheduler reserves what
|
||||
# it actually uses. Revisit with `kubectl top` once 26.8.2 has run for a week. Patched here (not in
|
||||
# the remote kubernetes.yml) so a Renovate bump keeps it.
|
||||
patches:
|
||||
- target:
|
||||
kind: Deployment
|
||||
name: keycloak-operator
|
||||
patch: |-
|
||||
- op: replace
|
||||
path: /spec/template/spec/containers/0/resources/limits/memory
|
||||
value: 768Mi
|
||||
- op: replace
|
||||
path: /spec/template/spec/containers/0/resources/requests/memory
|
||||
value: 512Mi
|
||||
+4
-6
@@ -142,9 +142,7 @@
|
||||
"description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)versions\\.env$/",
|
||||
"/(^|/)platform/pm-agent/.+\\.ya?ml$/",
|
||||
// ADDED 2026-11-08: the merge gate's job `container:` is the toolbox, by digest.
|
||||
"/^\\.gitea/workflows/gate\\.ya?ml$/"
|
||||
"/(^|/)platform/pm-agent/.+\\.ya?ml$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
|
||||
@@ -266,7 +264,7 @@
|
||||
// and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the
|
||||
// Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server
|
||||
// (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak:
|
||||
// 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
|
||||
// 26.0.6). The release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
|
||||
// markers under this ONE depName, so they move together in one branch (groupName makes that
|
||||
// explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every
|
||||
// app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6
|
||||
@@ -292,8 +290,8 @@
|
||||
"enabled": false
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids. `github-actions` ADDED 2026-11-08 for the same reason: it extracts the gate job's digest-only `container: image:` in .gitea/workflows/gate.yml, which the custom manager already follows through its marker.",
|
||||
"matchManagers": ["kubernetes", "github-actions"],
|
||||
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
|
||||
"matchManagers": ["kubernetes"],
|
||||
"matchPackageNames": [
|
||||
"gitea.home.keithjackson.dev/home-network/toolbox",
|
||||
"gitea.home.keithjackson.dev/home-network/agent-runner",
|
||||
|
||||
Reference in new issue
Block a user