jakman85andClaude Opus 5
a2440a3125
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
CI merge gate — .gitea/workflows/
Task T-0.2 · Requirements R-GITOPS-01, R-TEST-02, R-TEST-03 · Owner WS-GITOPS (CI).
This is the merge gate for the home k3s + Longhorn platform. It enforces the gating rules in the
spec (docs/14-testing-and-conformance.md,
ADR-0010,
ADR-0011) as a
Gitea Actions pipeline: no change lands on main unless the conformance subset for the requirements
it touches is green (R-TEST-03).
Done is a green, reproducible, requirement-tagged test — never "merged," never a demo. This gate is the machine that enforces it.
The pipeline
decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
live now cluster-gated (SKIPS in Phase 0)
| Stage | Job | Layer | Verifies | Cluster? |
|---|---|---|---|---|
| yaml/lint | lint |
— | YAML well-formed (ci/yamllint.yaml) |
no — live now |
| schema | l0 |
L0 | ServiceDefinitions valid against schemas/ |
no — live now |
| Kyverno L0 | l0 |
L0 | kyverno test platform/kyverno/: default-deny baseline present, no plaintext secrets, required labels/egress-class, schema conformance |
no — live now |
| L1 rendering | l1 |
L1 | ServiceDefinition → rendered resources match golden expectations (Chainsaw L1 via conformance/) |
no — live now |
| conformance | conformance |
L2+L3 | Real runtime behavior + fault-injection, as an in-cluster Chainsaw Job CI triggers and awaits | yes — cluster-gated |
| decision | merge-gate |
— | Applies R-TEST-03 over the above; blocks merge on any failure | no |
Each stage is a thin wrapper over a ci/*.sh script, so CI and a laptop run the identical gate
(R-TEST-02). See ../../ci/README.md.
Why L2/L3 is an in-cluster Job (not run by CI directly)
Per docs/14 §"Where each layer runs"
and ADR-0011: CI holds no broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
scoped test ServiceAccount holds cluster access (R-SEC-02). CI only triggers the Job and
collects its artifacts, using a narrow trigger credential referenced by secret name only
(secrets.CONFORMANCE_TRIGGER_KUBECONFIG) — never a plaintext secret, never a cluster-admin kubeconfig.
Honest SKIP — never fake green
NOTE (2026-09-29): this section is written in the past's voice — "Phase 0", "no cluster exists". A cluster has existed for weeks and Flux has been reconciling against it. The skip logic below is still correct and still honest, but read it as describing a live coverage gap (L2/L3 is not wired into CI) rather than an expected phase. Its being phrased as a phase is part of why the gap went unexamined.
Originally: The gate is deliberately built so unavailable checks SKIP loudly, they do not pass:
- The
conformancejob isif: ${{ vars.CLUSTER_AVAILABLE == 'true' }}— with no cluster it is skipped by Gitea (shown as skipped, not green). Themerge-gatedecision then reports any touched requirement that needs L2/L3 as UNVERIFIED, not satisfied. - Stages whose owning subtree is still a placeholder (
schemas/,platform/kyverno/,conformance/) report SKIPPED — unwired, distinct from a pass, until those seams are delivered.
The decision matrix (scripts/decide.sh):
| Situation | Outcome |
|---|---|
| Any of lint/l0/l1 fails | BLOCK |
| conformance fails/cancelled | BLOCK |
conformance skipped while CLUSTER_AVAILABLE=true |
BLOCK (it should have run) |
| conformance skipped, no cluster, touched reqs need L2/L3 | UNVERIFIED — reported, not green; blocks only if REQUIRE_CLUSTER_TESTS=true |
| lint/l0/l1 green, no cluster-needing reqs touched | PASS |
REQUIRE_CLUSTER_TESTS is false in Phase 0 (bootstrap: seams and cluster don't exist yet) and flips
to true from Phase 1 once a dev/test cluster is declared — from then on, any touched requirement that
can't be verified is a hard block.
Artifacts (R-TEST-04 traceability)
Every layer emits machine-readable output collected by the merge-gate:
- JUnit XML →
results/junit/*.xml(uploaded asl1-junit-coverage/conformance-junit-coverage). - Requirement-coverage report →
reports/coverage/**— maps everyR-*to its verifying test(s); uncovered requirements are visible gaps. Produced by the conformance runner (conformance/, WS-TEST).
Both dirs are git-ignored (see .gitignore).
Triggers & branch model (CONTRIBUTING.md)
pull_request→main: PRs are the enforced merge gate intomain.push→mainonly. The per-branch prefix list was removed:act_runnerruns at capacity 1, so a branch push fires a redundant run that serialises ahead of the PR run for the same tree. No work onmain— and since 2026-09-29 that is enforced, not conventional:mainis a protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
Transition: COMPLETE (2026-09-29)
This section used to describe a migration as pending. It has happened. Gitea is live at
gitea.home.keithjackson.dev, the runner incluster-dind is registered and online, and
gate.yml runs on every PR.
What took a month to notice: the migration was recorded as done on 2026-08-31 without
anyone checking the platform. In reality main had no branch protection and zero
runners were registered until 2026-09-29. Every "the gate enforces this" statement in
between rested on an assertion nobody verified. PR #1 — the first pull request this
repository ever had — sat pending for 30.6 minutes because nothing existed to run it.
So: do not read this file, or the DECISIONS-LOG, as evidence about the live platform.
Query the API. infra/gitea/ holds the scripts and the measured state.
Still genuinely outstanding: vars.CLUSTER_AVAILABLE and
secrets.CONFORMANCE_TRIGGER_KUBECONFIG are unset, so L2/L3 conformance does not run in
CI. That is a real coverage gap, not a phase.
Configuration (set in Gitea; referenced by name only)
| Kind | Name | Purpose |
|---|---|---|
| variable | CLUSTER_AVAILABLE |
'true' once a dev/test cluster exists → L2/L3 run and are required; else they SKIP |
| variable | REQUIRE_CLUSTER_TESTS |
'true' (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
| secret | CONFORMANCE_TRIGGER_KUBECONFIG |
narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
Files
.gitea/workflows/
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
scripts/
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
decide.sh the R-TEST-03 merge decision over job results
README.md this file
The stage logic itself lives in ../../ci/ so it is reusable locally and in CI.