Compare commits
76
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
72a46ad82d | ||
|
|
cadf64a514 | ||
|
|
cfbc596b8b | ||
|
|
c59a4b76bb | ||
|
|
cd2617e5b3 | ||
|
|
8697702d51 | ||
|
|
d29a73cad8 | ||
|
|
07f03536b3 | ||
|
|
e993d0f7d1 | ||
|
|
20ea70d221 | ||
|
|
8c7a770de2 | ||
|
|
1c5974013e | ||
|
|
4791d4a43a | ||
|
|
dfd608ac32 | ||
|
|
1d2d9d6549 | ||
|
|
4b31f97486 | ||
|
|
50a48e6f07 | ||
|
|
911ead5fbd | ||
|
|
8eb5baffa4 | ||
|
|
4634b44cc7 | ||
|
|
6586919132 | ||
|
|
e6f6e4febb | ||
|
|
e8e6b9c681 | ||
|
|
fcc66c7858 | ||
|
|
b9e8a4f9e8 | ||
|
|
b2d9883958 | ||
|
|
00efd4b828 | ||
|
|
67a54a33a4 | ||
|
|
db1a623073 | ||
|
|
ce271225de | ||
|
|
e72050c690 | ||
|
|
761e25cc3f | ||
|
|
e7508931ca | ||
|
|
bf98a99af6 | ||
|
|
8e017fad91 | ||
|
|
27d0ef3324 | ||
|
|
bbd4f682a6 | ||
|
|
0391139240 | ||
|
|
e9f441b5ce | ||
|
|
234cc6a71a | ||
|
|
69a0d1a1c7 | ||
|
|
d3a0d587a4 | ||
|
|
fe862ab527 | ||
|
|
a2665c0511 | ||
|
|
5a9e455149 | ||
|
|
4fd75ccdaa | ||
|
|
d7f24d0a9e | ||
|
|
712f4c9336 | ||
|
|
e3e56af461 | ||
|
|
a3f4ddb971 | ||
|
|
5f3ea63b7f | ||
|
|
23c335257b | ||
|
|
9339d5a79e | ||
|
|
a1cc0a5556 | ||
|
|
6eaaaae334 | ||
|
|
cde02d19d2 | ||
|
|
05a8094785 | ||
|
|
d99d47494c | ||
|
|
6f231a4311 | ||
|
|
623fa14256 | ||
|
|
16cbd4a79e | ||
|
|
755c08ab40 | ||
|
|
b389566641 | ||
|
|
7887ecbbdc | ||
|
|
d717d8e979 | ||
|
|
bc106caac1 | ||
|
|
79c703bc79 | ||
|
|
3523617671 | ||
|
|
6a4dc0ada7 | ||
|
|
14d5ccbadd | ||
|
|
05881d5a6a | ||
|
|
cf0d6c6beb | ||
|
|
20a28d0993 | ||
|
|
b184a42729 | ||
|
|
4080065aa7 | ||
|
|
16c6234696 |
No files matched your search
+29
-15
@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
|
||||
## The pipeline
|
||||
|
||||
```
|
||||
lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
|
||||
decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
|
||||
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
|
||||
live now cluster-gated (SKIPS in Phase 0)
|
||||
```
|
||||
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
|
||||
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
|
||||
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
|
||||
|
||||
## Honest SKIP — never fake green (Phase 0)
|
||||
## Honest SKIP — never fake green
|
||||
|
||||
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
|
||||
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
|
||||
skip logic below is still correct and still honest, but read it as describing a **live
|
||||
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
|
||||
phrased as a phase is part of why the gap went unexamined.
|
||||
|
||||
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
they do not pass**:
|
||||
|
||||
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
|
||||
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
|
||||
## Triggers & branch model (CONTRIBUTING.md)
|
||||
|
||||
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
|
||||
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
|
||||
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
|
||||
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
|
||||
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
|
||||
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
|
||||
|
||||
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
|
||||
## Transition: COMPLETE (2026-09-29)
|
||||
|
||||
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
|
||||
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
|
||||
This section used to describe a migration as pending. It has happened. Gitea is live at
|
||||
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
|
||||
`gate.yml` runs on every PR.
|
||||
|
||||
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
|
||||
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
|
||||
local feature-branch → `main`, the gate enforced by convention + the local runner.
|
||||
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
|
||||
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
|
||||
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
|
||||
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
|
||||
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
|
||||
anyone checking the platform. In reality `main` had **no branch protection** and **zero
|
||||
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
|
||||
between rested on an assertion nobody verified. PR #1 — the first pull request this
|
||||
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
|
||||
|
||||
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
|
||||
Query the API. `infra/gitea/` holds the scripts and the measured state.
|
||||
|
||||
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
|
||||
CI. That is a real coverage gap, not a phase.
|
||||
|
||||
## Configuration (set in Gitea; referenced by name only)
|
||||
|
||||
|
||||
@@ -44,16 +44,31 @@
|
||||
---
|
||||
name: gate
|
||||
|
||||
# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
|
||||
#
|
||||
# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
|
||||
# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
|
||||
#
|
||||
# * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
|
||||
# SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
|
||||
# clock on the one result that actually gates the merge. That cost real time during the
|
||||
# 2026-09-29 plex claim, where the loop is racing a short-lived token.
|
||||
# * The list silently did not match this repo's own convention. Branches here are `feat/**`,
|
||||
# which was absent, so the "fast feedback" never fired for the most common prefix anyway —
|
||||
# a mismatch nobody noticed for a month because its only symptom was a MISSING run.
|
||||
# Any hand-maintained prefix list has that failure mode; deleting it removes the class.
|
||||
# * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
|
||||
# you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
|
||||
# same checks buys nothing.
|
||||
#
|
||||
# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
|
||||
# which no PR context exists.
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- 'feature/**'
|
||||
- 'fix/**'
|
||||
- 'docs/**'
|
||||
- 'chore/**'
|
||||
|
||||
concurrency:
|
||||
group: gate-${{ github.ref }}
|
||||
@@ -95,9 +110,27 @@ jobs:
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
|
||||
# OWN database (branch protection + runner registration) — the blind spot that let `main`
|
||||
# sit unprotected with zero runners for a month while this repo recorded both as working.
|
||||
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
|
||||
#
|
||||
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
|
||||
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
|
||||
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
|
||||
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
|
||||
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
|
||||
# tolerance narrows back to spec-immutability alone.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
|
||||
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
|
||||
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
|
||||
# skipping, this line turns that into a failure and the PR goes red instead of
|
||||
# merging. Green here means the token resolved and the live Gitea matched the
|
||||
# declaration in infra/gitea/expected-branch-protection.json.
|
||||
REQUIRE_GITEA_SETTINGS: "true"
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
|
||||
@@ -11,11 +11,14 @@
|
||||
# v
|
||||
# merge-gate decision [R-TEST-03: touched subset green]
|
||||
#
|
||||
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
|
||||
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
|
||||
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
|
||||
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
|
||||
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
|
||||
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
|
||||
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
|
||||
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
|
||||
# as a plan and never revisited once the plan completed.
|
||||
#
|
||||
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
|
||||
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
|
||||
# statement about the live platform; query the API instead (see infra/gitea/).
|
||||
#
|
||||
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
|
||||
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
|
||||
# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
|
||||
# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
|
||||
# merely left alone. That is a one-way, outward-facing operation against a live authoritative
|
||||
# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
|
||||
# reconcile removes can be restored by hand from this file. It is the same discipline as
|
||||
# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
|
||||
# change rather than to a migration.
|
||||
#
|
||||
# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
|
||||
# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
|
||||
# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
|
||||
# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
|
||||
# devcontainer, which has previously produced false "no records" readings — the API is the only
|
||||
# reading that is both complete and authoritative.
|
||||
#
|
||||
# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
|
||||
# TTL and the proxy flag, not just the value.
|
||||
#
|
||||
# <type>|<name>|<content>|<ttl>|<proxied>
|
||||
#
|
||||
# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
|
||||
# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
|
||||
#
|
||||
# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
|
||||
# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
|
||||
# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
|
||||
# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
|
||||
# cannot touch them regardless of what this branch does:
|
||||
#
|
||||
# *.keithjackson.dev -> 136.51.115.252 legacy stack (the zone wildcard)
|
||||
# brickbreak.keithjackson.dev -> 20.236.242.14 unrelated, off-prem
|
||||
# dev.brickbreak… -> 20.236.242.14 unrelated, off-prem
|
||||
#
|
||||
# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
|
||||
# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
|
||||
# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
|
||||
# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
|
||||
# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
|
||||
# end state for `internal` services and it is also precisely why Pi-hole must already be
|
||||
# answering the LAN authoritatively before this branch merges.
|
||||
#
|
||||
# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
|
||||
# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
|
||||
# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
|
||||
# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
|
||||
# HTTPRoute that already had a published record produced
|
||||
# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
|
||||
# interval (~60s) later, and nothing else in the zone moved.
|
||||
#
|
||||
# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
|
||||
# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
|
||||
# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
|
||||
# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
|
||||
# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
|
||||
# points the registry at a source object that does not exist.
|
||||
A|*.keithjackson.dev|136.51.115.252|1|false
|
||||
A|auth.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
|
||||
A|crafty.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|gitea.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|grafana.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|jackett.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|pihole.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|plex.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|radarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|seerr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|unifi.home.keithjackson.dev|192.168.2.21|1|false
|
||||
TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
|
||||
TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
|
||||
TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
|
||||
TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
|
||||
TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
|
||||
TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
|
||||
TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
|
||||
TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
|
||||
TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
|
||||
TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
|
||||
TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
|
||||
TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
|
||||
TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
|
||||
TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
|
||||
TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
|
||||
TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false
|
||||
@@ -0,0 +1,171 @@
|
||||
# CLAUDE.md — how work lands in this repo
|
||||
|
||||
Read this before doing anything durable. It is the delivery contract for AI coding
|
||||
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
|
||||
Every durable change reaches `main` exactly one way:
|
||||
|
||||
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
|
||||
|
||||
There is no other path. Specifically:
|
||||
|
||||
- **Do not `git push origin main`.**
|
||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||
- **Do not merge your own PR.** A human approves and clicks merge.
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
|
||||
> That wording is **stale** — it predates the PR flow. It describes what Flux does
|
||||
> (main is what the cluster follows), not what you are permitted to do. Honour this
|
||||
> file, not that line. Fixing those runbooks is tracked work.
|
||||
|
||||
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
|
||||
|
||||
Verified against the live Gitea on 2026-09-29:
|
||||
|
||||
| Expected | Actual |
|
||||
|---|---|
|
||||
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
|
||||
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
|
||||
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
|
||||
| Approval required before merge | ✅ `required_approvals: 1` |
|
||||
|
||||
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
true. They are true now because each was measured, and the push rejection was provoked rather
|
||||
than assumed. **Re-measure before relying on any of them** — see §2b.
|
||||
|
||||
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
|
||||
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
|
||||
|
||||
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||
|
||||
## 2b. Never infer platform state from this repository
|
||||
|
||||
**This repo is not evidence about the live platform.** Neither is the handoff log
|
||||
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
|
||||
believed at the time of writing, and both have been confidently wrong for a month at a
|
||||
stretch.
|
||||
|
||||
On 2026-09-29 the log stated the merge gate was a required status check with a runner
|
||||
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
|
||||
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
|
||||
was wrong, and nothing in the tree could have revealed it, because the tree was the source
|
||||
of the error.
|
||||
|
||||
**So, before asserting that any gate, protection, runner, secret, or workload is in a
|
||||
given state: query the thing itself.**
|
||||
|
||||
| Claim about... | Check |
|
||||
|---|---|
|
||||
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
|
||||
| a runner exists | `GET /api/v1/admin/actions/runners` |
|
||||
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
|
||||
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
|
||||
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
|
||||
|
||||
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
|
||||
"unverified" is a legitimate report, and a far better one than a confident guess that
|
||||
matches the docs.
|
||||
|
||||
**A green tick is not proof anything ran.** Read what actually executed.
|
||||
|
||||
## 3. Gate before you push
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
|
||||
Exit codes are a contract — do not flatten them:
|
||||
|
||||
| rc | Meaning |
|
||||
|---|---|
|
||||
| 0 | PASS |
|
||||
| 1 | BLOCKED — something failed |
|
||||
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
|
||||
|
||||
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
|
||||
beside this one; that single skip is expected and tolerated. Any *other* skip is a
|
||||
real gap in your run.
|
||||
|
||||
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
|
||||
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
|
||||
|
||||
### Running the gate from a git worktree
|
||||
|
||||
Two unrelated failures, both with known workarounds — put both in any agent brief:
|
||||
|
||||
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
|
||||
and exists only in the main checkout. Do **not** copy it (it is a credential).
|
||||
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
|
||||
normally.
|
||||
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
|
||||
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
|
||||
the `ENGINE` hook — a shim that intercepts `run` and re-execs
|
||||
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
|
||||
`tracked_files()` is this bug, not a broken tree.
|
||||
|
||||
## 4. Branch naming
|
||||
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
|
||||
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
|
||||
|
||||
## 5. Cluster access
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the
|
||||
pinned container. Always name the context explicitly:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
|
||||
default has already changed once. Never rely on it. Do not run two `dev.sh`
|
||||
invocations concurrently.
|
||||
|
||||
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
|
||||
|
||||
## 6. Secrets
|
||||
|
||||
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
|
||||
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
|
||||
list key names or object names only. A password has been leaked into a transcript
|
||||
this way before.
|
||||
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
|
||||
the single creation-rules file; do not add a second one in a subtree.
|
||||
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
|
||||
flow. Do not design around committing one — read the expiry from the issuing page and
|
||||
treat a burnt token as free to retry.
|
||||
|
||||
## 7. Before you finish — write a handoff entry, in its OWN file
|
||||
|
||||
Add **one new file**:
|
||||
|
||||
decisions/YYYY-MM-DD-<slug>.md <slug> = your branch's last segment
|
||||
|
||||
with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
|
||||
|
||||
## What changed
|
||||
## What I verified
|
||||
## What I did NOT verify
|
||||
|
||||
State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
|
||||
and this repo has already been burned by the difference. All three headings are
|
||||
**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
|
||||
Template and full rationale: `decisions/README.md`.
|
||||
|
||||
**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
|
||||
SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
|
||||
to it. Appending to its tail is what every agent used to do, and because every branch
|
||||
appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
|
||||
on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
|
||||
`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
|
||||
anything to do with each other. Separate files have nothing to merge, which is the
|
||||
entire fix. Note that the damage from getting this wrong lands on *other people's*
|
||||
branches, later, which is why it is a gate stage and not a request.
|
||||
|
||||
Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
|
||||
wording is **stale**, in the same way §1's `git push origin main` wording is stale.
|
||||
+1548
File diff suppressed because it is too large.
Load diff
@@ -95,8 +95,22 @@ spec:
|
||||
disk:
|
||||
bus: virtio
|
||||
interfaces:
|
||||
# MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
|
||||
# KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
|
||||
# /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
|
||||
# that file while the NoCloud instance-id is unchanged — and the instance-id here is
|
||||
# domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
|
||||
# handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
|
||||
# administratively DOWN with no address and no DHCP attempt.
|
||||
# That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
|
||||
# device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
|
||||
# (because the per-instance modules were skipped), yet UOS is installed and
|
||||
# qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
|
||||
# Pinning the MAC to the value the current VMI already holds makes the guest's netplan
|
||||
# match permanently, so this cannot recur on the next recreation.
|
||||
- name: default
|
||||
masquerade: {}
|
||||
macAddress: a6:d7:c0:b3:72:fa
|
||||
# Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
|
||||
# these and (b) surfaces them as containerPorts on the virt-launcher pod so the
|
||||
# LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
|
||||
|
||||
@@ -5,8 +5,8 @@ metadata:
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
installer-url: ENC[AES256_GCM,data:CwdpCy4dKbJa7A==,iv:zdKrFad7Lg2g7bM7ALLPzp/3VIjXCahZABEscpJCYF4=,tag:inlLeAKrPrt9qvr8/Y4Hiw==,type:str]
|
||||
console-password: ENC[AES256_GCM,data:F+3lMUjA092RVg==,iv:h7NenRZUEJxTjzRDGCLBMHRtLl9h122wLvUvNGfmB50=,tag:Emo5y0qOUBptnXK03sgadQ==,type:str]
|
||||
installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
|
||||
console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -16,14 +16,14 @@ sops:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLMHpXYVhkblBNQWJ2NlZW
|
||||
WXJPSDcxUEdYakN0R1Nydkc1a2E5anJSNEV3CmM1MUpiNzZUa0dKQXNRZGpTRm9l
|
||||
UnMvMTNDL3gvUG1jVFdBbUI3aU01RkkKLS0tIGh2OUNyZzFpMXI1SWFSZENPNVVJ
|
||||
R2V4N3plREx5UGJ2emprTS9WRzljRXMKCWIHr3+L+H2QxocHmA5oDdjZBfvo2Z5F
|
||||
6JLNGs0M7O0X5n+IUqyM+pU+BbqHX+y2hTOSwOgCSV9Zw+ME1kUTKg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
|
||||
WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
|
||||
TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
|
||||
cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
|
||||
kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-14T07:13:55Z"
|
||||
mac: ENC[AES256_GCM,data:A2JQR++tYY9XrRmN1x24klU4xpanx0Ab64AIPGfvtHUxwtG2JLCMMAHIe64eM60FIKT5ibmnVVeuRIUoDwm5d3WpdX2gmGpzxyr9TDhEcEA2ENQGP5TydnmcmchNcu//FfXTXTgm91+WjBke06gibKEF9H+65s2YbkOtheFUjqA=,iv:tQkW7xDGMb8+35QAx+svTQDEwlvW8AQfouopeRqtNTg=,tag:fLJ58ovDbf5qy0sxBqUfPw==,type:str]
|
||||
lastmodified: "2026-09-19T02:47:13Z"
|
||||
mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -268,6 +268,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -17,6 +17,7 @@ resources:
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-bootstrap-secret.sops.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
|
||||
@@ -232,6 +232,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
|
||||
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
|
||||
#
|
||||
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
|
||||
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
|
||||
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
|
||||
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||
# not prose, so the upstream conditions below are described in words.
|
||||
#
|
||||
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
|
||||
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
|
||||
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
|
||||
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
|
||||
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
|
||||
#
|
||||
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
|
||||
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
|
||||
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
|
||||
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
|
||||
#
|
||||
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
|
||||
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
|
||||
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
|
||||
#
|
||||
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||
# the same file extracted from the running `plexinc/pms-docker:latest` image):
|
||||
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
|
||||
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
|
||||
# README: "If server is already logged in, this parameter is ignored."
|
||||
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
|
||||
# empty body. curl's exit status is never checked, the sed for the authentication-token element
|
||||
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
|
||||
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
|
||||
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
|
||||
# It does NOT crashloop.
|
||||
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
|
||||
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
|
||||
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
|
||||
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
|
||||
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
|
||||
# by hand at claim time.
|
||||
#
|
||||
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
|
||||
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||
# Never decrypt this file into a shell, a log, or a shared terminal.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plex-claim
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
type: Opaque
|
||||
stringData:
|
||||
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
|
||||
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
|
||||
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
|
||||
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-29T06:31:16Z"
|
||||
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -132,6 +132,11 @@ spec:
|
||||
value: Etc/UTC
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
|
||||
@@ -344,6 +344,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -4,7 +4,13 @@
|
||||
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
|
||||
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
|
||||
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
|
||||
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
|
||||
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
|
||||
# Gitea API and writes it here via `sops set`, without the value ever being printed.
|
||||
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
|
||||
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
|
||||
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
|
||||
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
|
||||
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
@@ -14,7 +20,7 @@ metadata:
|
||||
app.kubernetes.io/name: act-runner
|
||||
type: Opaque
|
||||
stringData:
|
||||
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
|
||||
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -30,8 +36,8 @@ sops:
|
||||
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
|
||||
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-26T04:48:14Z"
|
||||
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
|
||||
lastmodified: "2026-09-29T05:48:35Z"
|
||||
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+69
-106
@@ -99,70 +99,79 @@ spec:
|
||||
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
||||
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
||||
automountServiceAccountToken: false
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
|
||||
#
|
||||
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
|
||||
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
|
||||
# running alongside them, and is terminated after them. kubelet holds `runner` until
|
||||
# this container's startupProbe passes, so the ordering is enforced by the kubelet
|
||||
# instead of hand-rolled shell in the runner.
|
||||
#
|
||||
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
|
||||
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
|
||||
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
|
||||
# depended on bare dollar-names surviving it untouched. There is now no shell here at
|
||||
# all, so there is nothing to escape or mis-escape.
|
||||
#
|
||||
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
|
||||
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
|
||||
# and only THEN begins listening, so the port can accept a connection while the certs
|
||||
# are absent or incomplete -- and act_runner hard-exits on a missing
|
||||
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
|
||||
# are usable and that the daemon answers.
|
||||
#
|
||||
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
|
||||
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
|
||||
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
|
||||
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
|
||||
# attempt, landing microseconds after dockerd begins listening and cut off by
|
||||
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
|
||||
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
|
||||
# startup line would slow every subsequent poll. Left documented rather than silenced.
|
||||
#
|
||||
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
|
||||
initContainers:
|
||||
- name: dind
|
||||
restartPolicy: Always # <- this field is what makes it a SIDECAR
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
- docker
|
||||
- --host=tcp://127.0.0.1:2376
|
||||
- --tlsverify
|
||||
- --tlscacert=/certs/client/ca.pem
|
||||
- --tlscert=/certs/client/cert.pem
|
||||
- --tlskey=/certs/client/key.pem
|
||||
- info
|
||||
periodSeconds: 1
|
||||
failureThreshold: 60
|
||||
containers:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
|
||||
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
|
||||
# that used to live here are gone: kubelet now holds this container until the dind
|
||||
# sidecar's startupProbe passes (see initContainers above).
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
@@ -198,52 +207,6 @@ spec:
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
|
||||
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -227,6 +227,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
+46
-9
@@ -32,11 +32,46 @@ env:
|
||||
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
||||
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
||||
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
||||
# single API POST) against the running server; the claim then persists on the /config PVC and
|
||||
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
||||
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
|
||||
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
|
||||
# workload). Plex cannot create libraries over the mounted bulk share while the server is
|
||||
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||
# half of that reasoning survived contact with the upstream source.
|
||||
#
|
||||
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
|
||||
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
|
||||
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
|
||||
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
|
||||
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
|
||||
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
|
||||
#
|
||||
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
|
||||
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
|
||||
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
|
||||
# committed value is the spent token from that claim rather than the original placeholder.
|
||||
#
|
||||
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||
# the running `:latest` image), NOT assumed:
|
||||
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
|
||||
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
|
||||
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
|
||||
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
|
||||
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
|
||||
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
|
||||
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
|
||||
# default 0, which continues startup even when a cont-init script fails.
|
||||
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
|
||||
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
|
||||
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
|
||||
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
|
||||
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
||||
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
||||
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
||||
@@ -56,10 +91,12 @@ storage:
|
||||
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
||||
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
||||
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
||||
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
||||
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
|
||||
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
|
||||
# exchange is then skipped on every later start. (The older note here claimed a
|
||||
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
|
||||
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
|
||||
# failure, and that is why one is committed.)
|
||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||
# downloads/ movies/ photos/ tv/).
|
||||
|
||||
+179
-2
@@ -80,6 +80,18 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
|
||||
BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
|
||||
GATEWAY_NAME = "shared"
|
||||
GATEWAY_NS = "gateway"
|
||||
# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
|
||||
# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
|
||||
# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
|
||||
# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
|
||||
# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
|
||||
# typo — "no DNS controller owns this route" — and because the one value that must never be
|
||||
# emitted here is external-dns's own, which a literal `none` cannot drift into.
|
||||
EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
EXTERNALDNS_CONTROLLER_DISOWN = "none"
|
||||
# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
|
||||
# assert the disown value is not it; never emitted.
|
||||
EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
|
||||
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
|
||||
MONITORING_NS = "monitoring" # where Prometheus runs; the only admitted scrape source
|
||||
|
||||
@@ -836,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list:
|
||||
|
||||
|
||||
def _route_meta(s: dict, route: dict) -> dict:
|
||||
"""HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
|
||||
`exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
|
||||
public DNS record, and now cannot)."""
|
||||
meta = {
|
||||
"name": f"{s['name']}-{route['port']}",
|
||||
"namespace": s["name"],
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
}
|
||||
ann = _auth_annotations(s)
|
||||
ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
|
||||
if ann:
|
||||
meta["annotations"] = ann
|
||||
return meta
|
||||
@@ -1039,6 +1054,64 @@ def _auth_annotations(s: dict) -> dict:
|
||||
return {}
|
||||
|
||||
|
||||
def _externaldns_route_annotations(s: dict) -> dict:
|
||||
"""ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
|
||||
|
||||
`internal` renders NO public DNS record. That sentence was in the contract from the start and
|
||||
nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
|
||||
`internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
|
||||
Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
|
||||
(jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
|
||||
subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
|
||||
of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
|
||||
|
||||
THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
|
||||
`registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
|
||||
from memory. external-dns v0.21.0 `source/annotations/processors.go`:
|
||||
|
||||
func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
|
||||
value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
|
||||
if ok && value != ControllerValue { ...; return true }
|
||||
return false
|
||||
}
|
||||
|
||||
with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
|
||||
`ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
|
||||
`Endpoints()`, BEFORE any hostname is resolved:
|
||||
|
||||
if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
||||
|
||||
so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
|
||||
endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
|
||||
so the same annotation scopes the `service` source too if a rendered Service ever grows an
|
||||
external-dns hostname.
|
||||
|
||||
WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
|
||||
with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
|
||||
service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
|
||||
`catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
|
||||
`exposure: internal` to `public` is the only edit that can publish a name, which is exactly
|
||||
the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
|
||||
conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
|
||||
every gate run, including that this function reads `exposure` rather than naming services.
|
||||
|
||||
WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
|
||||
v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
|
||||
source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
|
||||
what is labelled" posture is available and is STRONGER by default. It is not taken here
|
||||
because its blast radius is not this defect's: seven published names come from hand-authored
|
||||
platform/apps manifests that carry no `exposure` field and that this renderer never sees
|
||||
(auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
|
||||
opt-in filter DELETES all seven the moment it is set, and whether each should stay published
|
||||
is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
|
||||
radius of the reported defect. The filter remains the available hardening once those seven
|
||||
are adjudicated — see platform/externaldns/helmrelease.yaml.
|
||||
"""
|
||||
if s["exposure"] == "internal":
|
||||
return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
|
||||
return {}
|
||||
|
||||
|
||||
def r_servicemonitor(s: dict):
|
||||
"""ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
|
||||
|
||||
@@ -1771,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
convention:
|
||||
- `<svc>.yaml` -> wave(svc)
|
||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
|
||||
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
|
||||
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
|
||||
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
|
||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||
@@ -1797,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
w = waves[name]
|
||||
add(w, f"{name}.yaml")
|
||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
|
||||
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
|
||||
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
|
||||
# Placed only when the file exists, so services without one are unaffected.
|
||||
add(w, f"{name}-claim-secret.sops.yaml")
|
||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||
@@ -2097,6 +2179,96 @@ def _selftest() -> int:
|
||||
"a non-exposed service must not render a gateway-ingress allow (least privilege)",
|
||||
)
|
||||
|
||||
# 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
|
||||
# public DNS record and must be INCAPABLE of rendering one: its route carries
|
||||
# external-dns's own controller annotation set to something other than `dns-controller`,
|
||||
# which v0.21.0's gateway-httproute source (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch) uses to skip the route before any hostname is
|
||||
# resolved. `public` must NOT carry it, or the five public names would vanish instead.
|
||||
# Both directions are asserted here because only asserting the internal half would pass
|
||||
# just as happily if the renderer stamped EVERY route and silently unpublished the zone.
|
||||
dns_int_docs = render(
|
||||
{
|
||||
"name": "di",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(
|
||||
len(dns_int_routes) == 1,
|
||||
"an internal exposed service must still render its HTTPRoute (LAN reachability is "
|
||||
"unchanged — only the PUBLIC record is withdrawn)",
|
||||
)
|
||||
if dns_int_routes:
|
||||
iann = dns_int_routes[0]["metadata"].get("annotations", {})
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
|
||||
)
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
|
||||
"the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
|
||||
"the route rather than releasing it — the one value that inverts the guarantee)",
|
||||
)
|
||||
dns_pub_docs = render(
|
||||
{
|
||||
"name": "dp",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "public",
|
||||
"ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "native"},
|
||||
}
|
||||
)
|
||||
dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
|
||||
if dns_pub_routes:
|
||||
check(
|
||||
EXTERNALDNS_CONTROLLER_KEY
|
||||
not in dns_pub_routes[0]["metadata"].get("annotations", {}),
|
||||
"a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
|
||||
)
|
||||
# The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
|
||||
# ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
|
||||
# field is the likeliest way a new service arrives, and it is the case a per-service
|
||||
# allow-list would miss.
|
||||
dns_dflt_docs = render(
|
||||
{
|
||||
"name": "dd",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
|
||||
== EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
|
||||
"unpublishable (default posture, renderer-contract)",
|
||||
)
|
||||
# The scoping annotation must COEXIST with the auth hints rather than replace them: both are
|
||||
# merged into one `annotations` map, and a forward-auth internal app is the overlap case.
|
||||
dns_fa_docs = render(
|
||||
{
|
||||
"name": "df",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
|
||||
}
|
||||
)
|
||||
dns_fa_ann = next(
|
||||
d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
|
||||
)["metadata"]["annotations"]
|
||||
check(
|
||||
dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
|
||||
and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
|
||||
"DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
|
||||
)
|
||||
|
||||
# 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
|
||||
# PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
|
||||
# declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
|
||||
@@ -2385,7 +2557,8 @@ def _selftest() -> int:
|
||||
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
||||
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
||||
present = {
|
||||
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"a.yaml", "a-claim-secret.sops.yaml",
|
||||
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
||||
"svc-db-backup-credentials-prod.sops.yaml",
|
||||
"gitea.yaml",
|
||||
@@ -2396,6 +2569,10 @@ def _selftest() -> int:
|
||||
plan = plan_apps_waves(plan_sds, present)
|
||||
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||
check(flat.get("a-claim-secret.sops.yaml") == 0,
|
||||
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
|
||||
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
|
||||
"CreateContainerConfigError")
|
||||
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
||||
|
||||
+7
-1
@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
|
||||
help: ## Show this help
|
||||
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
|
||||
@@ -29,9 +29,15 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
|
||||
render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
|
||||
@"$(CI_DIR)render-golden.sh"
|
||||
|
||||
decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
|
||||
@"$(CI_DIR)decisions-log.sh"
|
||||
|
||||
l1: ## L1 rendering checks (no cluster; live now)
|
||||
@"$(CI_DIR)l1.sh"
|
||||
|
||||
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
||||
@./gitea-settings.sh
|
||||
|
||||
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
|
||||
@"$(CI_DIR)conformance.sh"
|
||||
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# `ci/` — portable merge gate (runnable today, without Gitea)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
||||
|
||||
The same merge gate the Gitea Actions workflow runs
|
||||
@@ -87,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|
||||
|---|---|---|
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
||||
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
|
||||
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
||||
|
||||
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
|
||||
|
||||
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
||||
#
|
||||
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
||||
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
||||
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
||||
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
||||
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
||||
#
|
||||
# MEASURED 2026-09-29, before this guard existed:
|
||||
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
||||
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
||||
# "[ ok ] L2/L3 conformance passed for touched subset"
|
||||
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
||||
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
||||
# it because nobody had ever set the variable.
|
||||
#
|
||||
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
||||
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
||||
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
||||
# makes the stage itself honour that instead of reporting a pass.
|
||||
JUNIT="${CONF_ART}/report.junit.xml"
|
||||
if [ -f "$JUNIT" ]; then
|
||||
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
||||
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
||||
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
||||
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
||||
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
||||
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
||||
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
||||
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
||||
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
||||
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
||||
exit 1
|
||||
fi
|
||||
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
||||
fi
|
||||
|
||||
ok "L2/L3 conformance passed for touched subset"
|
||||
summary "- conformance-l2-l3: **passed**"
|
||||
Executable
+195
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS
|
||||
# ---------------------
|
||||
# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
|
||||
# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
|
||||
# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
|
||||
# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
|
||||
# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
|
||||
#
|
||||
# #22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
|
||||
# #20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
|
||||
# #19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
|
||||
# #18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
|
||||
# #17 feat/ci-is-the-approver clean — it wrote no handoff entry
|
||||
#
|
||||
# Nothing was wrong with any of those changes. The log's storage format serialised work that
|
||||
# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
|
||||
# rebases, each invalidating the gate run before it).
|
||||
#
|
||||
# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
|
||||
# disjoint paths and there is nothing to merge. See decisions/README.md.
|
||||
#
|
||||
# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
|
||||
# --------------------------------------------
|
||||
# The old habit is written into this repo in many places — the runbooks, the muscle memory of
|
||||
# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
|
||||
# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
|
||||
# that quietly re-creates the conflict for every other open branch, and the damage is only
|
||||
# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
|
||||
# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
|
||||
# it gets a guard rather than a paragraph.
|
||||
#
|
||||
# WHAT IS ASSERTED
|
||||
# (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
|
||||
# Re-pinning is a deliberate, reviewable line in a diff; appending is not.
|
||||
# (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
|
||||
# directory sorts chronologically and two agents cannot collide on a filename.
|
||||
# (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
|
||||
# filename — a disagreement means a copied template nobody edited.
|
||||
# (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
|
||||
# "What changed", "What I verified", "What I did NOT verify". This is the quality floor
|
||||
# that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
|
||||
# this repo's recurring failure is a confident report of something nobody measured, and an
|
||||
# entry with nowhere to write "UNVERIFIED" does not get one written.
|
||||
#
|
||||
# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
|
||||
# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
|
||||
# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
|
||||
# add a fail-open surface to a guard whose whole job is to close one.
|
||||
set -uo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
set +e # aggregate every problem in one run; do not stop at the first.
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
ARCHIVE="DECISIONS-LOG.md"
|
||||
PIN="decisions/ARCHIVE.sha256"
|
||||
rc=0
|
||||
|
||||
# ---- (1) the archive is frozen ----------------------------------------------
|
||||
log "Stage: decisions log (archive frozen + entries well-formed)"
|
||||
|
||||
if [ ! -f "$ARCHIVE" ]; then
|
||||
err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
|
||||
summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
|
||||
rc=1
|
||||
elif [ ! -f "$PIN" ]; then
|
||||
err "${PIN} is missing, so the archive's integrity cannot be established."
|
||||
err " If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
|
||||
summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
|
||||
rc=1
|
||||
else
|
||||
want="$(awk 'NR==1{print $1}' "$PIN")"
|
||||
got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
|
||||
if [ "$want" != "$got" ]; then
|
||||
err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
|
||||
err " pinned: ${want}"
|
||||
err " on disk: ${got}"
|
||||
err ""
|
||||
err " Every branch that appends to this file's tail conflicts with every other branch that"
|
||||
err " does, on the same hunk. That is what made four of five open PRs unmergeable on"
|
||||
err " 2026-09-29, and your commit would re-create it for branches that are not yours."
|
||||
err ""
|
||||
err " Write your handoff as its OWN FILE instead:"
|
||||
err " decisions/\$(date +%F)-<your-branch-slug>.md (see decisions/README.md)"
|
||||
err " and restore the archive: git checkout -- ${ARCHIVE}"
|
||||
err ""
|
||||
err " If the archive genuinely must change (redacting a leaked credential, say), that is a"
|
||||
err " reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
|
||||
summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
|
||||
rc=1
|
||||
else
|
||||
ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- (2)(3)(4) every entry is well-formed ------------------------------------
|
||||
shopt -s nullglob
|
||||
entries=( decisions/[0-9]*.md )
|
||||
shopt -u nullglob
|
||||
|
||||
# Zero entries is legitimate only until the first handoff lands; after that the glob always
|
||||
# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
|
||||
# CLAUDE.md, not this stage, is what asks for one.
|
||||
if [ "${#entries[@]}" -eq 0 ]; then
|
||||
log "no decisions/ entries yet"
|
||||
fi
|
||||
|
||||
REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
|
||||
|
||||
for f in "${entries[@]}"; do
|
||||
base="${f#decisions/}"
|
||||
|
||||
# (2) filename
|
||||
if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
|
||||
err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
|
||||
err " Use your branch's last segment as the slug — it is already unique per unit of work."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fname_date="${BASH_REMATCH[1]}"
|
||||
|
||||
if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
|
||||
err "${f}: '${fname_date}' is not a real calendar date."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
|
||||
# (3) frontmatter
|
||||
if [ "$(head -1 "$f")" != "---" ]; then
|
||||
err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
|
||||
err " See decisions/README.md for the template."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
|
||||
missing_keys=()
|
||||
for k in date branch title; do
|
||||
grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
|
||||
done
|
||||
if [ "${#missing_keys[@]}" -gt 0 ]; then
|
||||
err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
|
||||
if [ "$fm_date" != "$fname_date" ]; then
|
||||
err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
|
||||
err " One of them was copied from a template and not edited; the log's ordering depends on both."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
|
||||
# (4) the three required sections, each with a real body
|
||||
body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
|
||||
[ -n "$body_start" ] || body_start=1
|
||||
for want_sec in "${REQUIRED_SECTIONS[@]}"; do
|
||||
hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
|
||||
if [ -z "$hdr_line" ]; then
|
||||
err "${f}: missing the required section '## ${want_sec}'."
|
||||
err " CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
|
||||
err " All three are structure here, because the third is the one that gets dropped."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
# A real body: at least 20 non-blank, non-heading characters before the next '## '.
|
||||
nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
|
||||
| tr -d '[:space:]' | wc -c)"
|
||||
if [ "$nchars" -lt 20 ]; then
|
||||
err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
|
||||
err " If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
|
||||
err " is a valid and useful entry. A blank section is not."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
|
||||
rc=1
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
|
||||
summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
|
||||
else
|
||||
err "decisions log stage FAILED"
|
||||
fi
|
||||
exit "$rc"
|
||||
+43
@@ -108,10 +108,12 @@ summary "## Merge gate (R-TEST-03)"
|
||||
summary ""
|
||||
|
||||
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
|
||||
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
|
||||
run_stage lint "${CI_DIR}/lint.sh"
|
||||
run_stage l0 "${CI_DIR}/l0.sh"
|
||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||
run_stage l1 "${CI_DIR}/l1.sh"
|
||||
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
||||
run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
|
||||
echo >&2
|
||||
@@ -188,6 +190,47 @@ case "${OUTCOME[spec-immutability]}" in
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
||||
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
||||
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
||||
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
|
||||
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
|
||||
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
|
||||
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
|
||||
# token is wired, and the skip becomes a block as it should.
|
||||
case "${OUTCOME[gitea-settings]}" in
|
||||
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
|
||||
skip)
|
||||
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
|
||||
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
|
||||
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
|
||||
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
|
||||
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
|
||||
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
|
||||
NOT_ESTABLISHED+=("gitea-settings")
|
||||
fi ;;
|
||||
pass) ok "gitea-settings: green" ;;
|
||||
esac
|
||||
|
||||
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
|
||||
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
|
||||
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
|
||||
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
|
||||
# the tree under test and needs no tool or cluster, so there is no environment in which it
|
||||
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
|
||||
# have a way to not-run.
|
||||
case "${OUTCOME[decisions-log]}" in
|
||||
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
|
||||
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
|
||||
err " as a defect in the stage itself and block."
|
||||
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
|
||||
BLOCK=1 ;;
|
||||
pass) ok "decisions-log: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
|
||||
Executable
+133
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
|
||||
# match what git declares.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS AT ALL
|
||||
# ----------------------------
|
||||
# Every other check in this repo reads the repository, the rendered manifests, or the
|
||||
# cluster. NONE of them can see Gitea's own database — and that is where branch
|
||||
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
|
||||
#
|
||||
# * `main` had NO branch protection, and
|
||||
# * ZERO Actions runners were registered,
|
||||
#
|
||||
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
|
||||
# and human in the loop repeated that claim because the repository said so. PR #1 — the
|
||||
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
|
||||
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
|
||||
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
|
||||
# merged with zero approvals recorded against `required_approvals: 1`.
|
||||
#
|
||||
# Three separate failures, one shape: a setting that READ as enforced while not being, with
|
||||
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
|
||||
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
|
||||
#
|
||||
# WHAT IT ASSERTS
|
||||
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
|
||||
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
|
||||
# 2. at least `.runners_online_min` Actions runners report status "online".
|
||||
#
|
||||
# (2) is not decoration. A required status check with no runner is the deadlock this repo
|
||||
# nearly walked into: every PR pends forever, including the one that would fix it.
|
||||
#
|
||||
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
|
||||
# `protected`, `enable_status_check` and `required_approvals` but NOT
|
||||
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
|
||||
# only account that merges. This stage queries /branch_protections/ deliberately.
|
||||
#
|
||||
# EXIT CODES (the gate contract)
|
||||
# 0 live Gitea matches the declaration
|
||||
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
|
||||
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
|
||||
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
|
||||
# that an unwired credential cannot lock the repository.
|
||||
#
|
||||
# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
|
||||
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
|
||||
# the drift it exists to catch is still undetected.
|
||||
#
|
||||
# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
|
||||
# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
|
||||
# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
|
||||
# detector sat inert for hours while a token existed and no secret did. The env var below still
|
||||
# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
|
||||
# the secret that would feed it cannot be created at all.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: Gitea platform settings match the declaration"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
|
||||
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
|
||||
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
|
||||
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
|
||||
# it looked like it had worked. A token was minted, no secret existed, and this stage would
|
||||
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
|
||||
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool curl
|
||||
|
||||
if [ -z "$TOKEN" ]; then
|
||||
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
|
||||
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
|
||||
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
|
||||
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
BRANCH="$(jq -r '.branch' "$DECL")"
|
||||
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
|
||||
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
|
||||
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
|
||||
skip " lacks permission). Not treating this as a pass."
|
||||
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
RC=0
|
||||
|
||||
# --- 1. every declared protection key must match the live rule, exactly ---------------
|
||||
while IFS= read -r key; do
|
||||
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
|
||||
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
|
||||
if [ "$want" = "$got" ]; then
|
||||
ok " ${key} = ${got}"
|
||||
else
|
||||
err " ${key}: DECLARED ${want} but LIVE is ${got}"
|
||||
RC=1
|
||||
fi
|
||||
done < <(jq -r '.protection | keys[]' "$DECL")
|
||||
|
||||
# --- 2. a runner must actually be online ----------------------------------------------
|
||||
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
|
||||
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
|
||||
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
|
||||
if [ "${N:-0}" -ge "$MIN" ]; then
|
||||
ok " runners online: ${N} (>= ${MIN})"
|
||||
else
|
||||
err " runners online: ${N}, DECLARED minimum ${MIN}."
|
||||
err " A required status check with no runner leaves every PR pending forever —"
|
||||
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
|
||||
RC=1
|
||||
fi
|
||||
|
||||
if [ "$RC" -eq 0 ]; then
|
||||
ok "Gitea settings match the declaration."
|
||||
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
|
||||
else
|
||||
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
|
||||
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
|
||||
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
|
||||
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
|
||||
fi
|
||||
exit "$RC"
|
||||
@@ -79,12 +79,37 @@ else
|
||||
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
||||
find "$l0_render" -type f -name '*.yaml' -print0 \
|
||||
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
||||
if kyverno test "$l0_render"; then
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
|
||||
# error, bad match block, malformed JMESPath) does not fail the run: every result row
|
||||
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
|
||||
#
|
||||
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
|
||||
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
|
||||
# rows came back `Skip / Invalid Policy`, and the summary read:
|
||||
# Test Summary: 9 tests passed and 0 tests failed
|
||||
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
|
||||
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
|
||||
#
|
||||
# This is not specific to that policy: a typo in restrict-host-namespaces or
|
||||
# secret-provenance would silently disable it and the gate would stay green. Scan the
|
||||
# output for the load failure the exit code hides.
|
||||
l0_kout="$(mktemp)"
|
||||
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
|
||||
if grep -q 'Invalid Policy' "$l0_kout"; then
|
||||
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
|
||||
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
|
||||
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
|
||||
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
|
||||
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
|
||||
rc=1
|
||||
else
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
fi
|
||||
else
|
||||
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
||||
fi
|
||||
rm -f "$l0_kout"
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
|
||||
@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
# --include-scope: union the touched subset with every L1 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
|
||||
# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
|
||||
# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
|
||||
# "someone must remember" is the failure mode this whole change removes; see
|
||||
# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
|
||||
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
|
||||
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
|
||||
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
|
||||
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
|
||||
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
|
||||
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
l1_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
|
||||
Executable
+130
@@ -0,0 +1,130 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
|
||||
#
|
||||
# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
|
||||
# for every other open branch. A guard against a silent regression is itself worth nothing
|
||||
# unless somebody has watched it reject the thing it claims to reject, so each case below
|
||||
# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
|
||||
# as ci/tests/gate-decision.test.sh, one layer down.
|
||||
set -uo pipefail
|
||||
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
|
||||
mk_tree() {
|
||||
local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
|
||||
cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
|
||||
printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
|
||||
( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
|
||||
mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
|
||||
# REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
|
||||
( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
|
||||
}
|
||||
|
||||
mk_entry() {
|
||||
cat > "$1" <<ENTRY
|
||||
---
|
||||
date: $2
|
||||
branch: chore/example
|
||||
title: An example entry
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
|
||||
## What I verified
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
ENTRY
|
||||
}
|
||||
|
||||
# run <dir> -> sets RC and OUT
|
||||
run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
|
||||
|
||||
D="${TMP}/t"
|
||||
|
||||
echo "== 1. a well-formed tree passes"
|
||||
mk_tree "$D"; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
|
||||
mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
|
||||
|| bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
|
||||
grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
|
||||
|| bad "the message does not say the archive is frozen"
|
||||
grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
|
||||
|| bad "the message does not name the remedy"
|
||||
|
||||
echo
|
||||
echo "== 3. a re-pin is a deliberate, accepted act"
|
||||
mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
|
||||
( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 4. a missing pin is unverifiable, not green"
|
||||
mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
|
||||
|
||||
echo
|
||||
echo "== 5. entry naming"
|
||||
for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
|
||||
mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
|
||||
done
|
||||
mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 6. frontmatter"
|
||||
mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
|
||||
|
||||
for key in date branch title; do
|
||||
mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
|
||||
done
|
||||
|
||||
# The copied-template failure: the body was rewritten, the date never was.
|
||||
mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
|
||||
|| bad "accepted a date/filename disagreement"
|
||||
|
||||
echo
|
||||
echo "== 7. the three required sections, and their bodies"
|
||||
for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
|
||||
mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
|
||||
done
|
||||
|
||||
# A heading with nothing under it is the way this floor gets defeated in practice.
|
||||
mk_tree "$D"
|
||||
python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
|
||||
import sys,re
|
||||
p=sys.argv[1]; s=open(p).read()
|
||||
s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
|
||||
open(p,'w').write(s)
|
||||
PY
|
||||
run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
|
||||
|
||||
echo
|
||||
echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
|
||||
mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -36,7 +36,13 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
|
||||
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
|
||||
|
||||
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
|
||||
for stage in spec-immutability lint l0 render-golden l1 conformance; do
|
||||
# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
|
||||
# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
|
||||
# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
|
||||
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
|
||||
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
|
||||
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
|
||||
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
|
||||
cat > "${CI}/${stage}.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
||||
@@ -184,6 +190,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
|
||||
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
|
||||
|
||||
echo
|
||||
echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
|
||||
# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
|
||||
# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
|
||||
# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
|
||||
# if one is ever reported it is a defect in the stage, and the gate must not soften it into
|
||||
# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
|
||||
# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
|
||||
run_gate STAGE_RC_decisions_log=1
|
||||
assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
|
||||
run_gate STAGE_RC_decisions_log=3
|
||||
assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
|
||||
run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
|
||||
assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -136,6 +136,65 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
||||
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
||||
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
|
||||
#
|
||||
# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
|
||||
# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
|
||||
# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
|
||||
# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
|
||||
# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
|
||||
# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
|
||||
#
|
||||
# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
|
||||
# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
|
||||
#
|
||||
# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
|
||||
# from the PATH and not from the content scan.
|
||||
|
||||
# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
|
||||
# under test here, not just the presence of both arms.
|
||||
git -C "$TMP" checkout -q -b feature-apps-vms main
|
||||
mkdir -p "$TMP/apps/vms/demo"
|
||||
printf 'kind: VirtualMachine\nmetadata:\n name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
|
||||
OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
|
||||
assert_present "R-PLAT-03" "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
|
||||
assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
|
||||
assert_absent "R-PLAT-02" "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
|
||||
|
||||
# A plain ServiceDefinition under apps/ gets the container-workload set.
|
||||
git -C "$TMP" checkout -q -b feature-apps main
|
||||
mkdir -p "$TMP/apps/wave-1"
|
||||
printf 'kind: ServiceDefinition\nmetadata:\n name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
|
||||
OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
|
||||
assert_present "R-PLAT-02" "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
|
||||
assert_present "R-NET-01" "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
|
||||
assert_present "R-NET-06" "$OUT_APPS" "apps/* attributes the egress-class requirements"
|
||||
assert_present "R-IDP-02" "$OUT_APPS" "apps/* attributes the auth-mode requirements"
|
||||
|
||||
# homepage / headlamp ARE the command-centre surface.
|
||||
git -C "$TMP" checkout -q -b feature-c2 main
|
||||
mkdir -p "$TMP/platform/headlamp"
|
||||
printf 'kind: Deployment\nmetadata:\n name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
|
||||
OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
|
||||
assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
|
||||
assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
|
||||
|
||||
# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
|
||||
git -C "$TMP" checkout -q -b feature-assembly main
|
||||
mkdir -p "$TMP/platform"
|
||||
printf 'resources:\n - cilium\n - longhorn\n' > "$TMP/platform/kustomization.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
|
||||
OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
|
||||
assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
|
||||
#
|
||||
|
||||
@@ -177,6 +177,23 @@ path_requirements() {
|
||||
case "$1" in
|
||||
catalog/*) echo "R-GITOPS-02" ;;
|
||||
schemas/*) echo "R-GITOPS-02" ;;
|
||||
# apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
|
||||
# manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
|
||||
# adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
|
||||
# checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
|
||||
# class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
|
||||
# violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
|
||||
# catch it at apply time" is post-merge, and the gate exists to catch it before.
|
||||
#
|
||||
# ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
|
||||
# apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
|
||||
# subject to default-deny (R-NET-01).
|
||||
apps/vms/*) echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
|
||||
# Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
|
||||
# workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
|
||||
# (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
|
||||
# default-deny (R-NET-01).
|
||||
apps/*) echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
|
||||
platform/kyverno/*) echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
|
||||
platform/cilium/*) echo "R-NET-01 R-NET-02 R-NET-05" ;;
|
||||
platform/longhorn/*) echo "R-STATE-01 R-STATE-02" ;;
|
||||
@@ -208,6 +225,39 @@ path_requirements() {
|
||||
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
# The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
|
||||
# rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
|
||||
# itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
|
||||
# docs/12-command-center-ui.md), so editing them must select the C2 tests.
|
||||
platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
|
||||
# KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
|
||||
platform/kubevirt/*) echo "R-PLAT-03 R-OBS-01" ;;
|
||||
# The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
|
||||
# TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
|
||||
platform/gateway/*) echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
|
||||
# CoreDNS is cluster DNS resolution (R-DNS-01).
|
||||
platform/coredns/*) echo "R-DNS-01" ;;
|
||||
# crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
|
||||
# its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
|
||||
platform/crafty/*) echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
|
||||
# Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
|
||||
platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
|
||||
# runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
|
||||
# resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
|
||||
# is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
|
||||
# runbook change is an observability change.
|
||||
runbooks/*) echo "R-OBS-04" ;;
|
||||
# platform-config/ is the env-specific configuration Flux substitutes into and applies.
|
||||
platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
|
||||
# platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
|
||||
# Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
|
||||
# file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
|
||||
# change. That is the most consequential single-line edit in the repo and it attributed nothing.
|
||||
platform/kustomization.yaml) echo "R-GITOPS-01" ;;
|
||||
# .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
|
||||
# here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
|
||||
.sops.yaml) echo "R-SEC-01 R-GITOPS-01" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
|
||||
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# clusters/
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
||||
|
||||
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
|
||||
#
|
||||
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
|
||||
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
|
||||
# meant to verify is worse than no harness.
|
||||
#
|
||||
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
|
||||
# waves needs it.
|
||||
#
|
||||
# NOTE: this reconciles RBAC that is INERT until
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
|
||||
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
|
||||
# trigger identity can exist while the policy bounding it does not.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: conformance
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./conformance/incluster
|
||||
@@ -33,6 +33,7 @@ resources:
|
||||
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
||||
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
||||
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
||||
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
|
||||
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
||||
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
||||
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
||||
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
||||
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# In-cluster conformance Job (L2/L3)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
||||
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
||||
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
|
||||
#
|
||||
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
|
||||
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
|
||||
# reconciling it would create a permanent Job object and fight the trigger for ownership.
|
||||
#
|
||||
# The identities here are inert without
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
|
||||
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
|
||||
# half — see the header of rbac.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- rbac.yaml
|
||||
@@ -174,3 +174,68 @@ subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-runner
|
||||
namespace: conformance
|
||||
---
|
||||
# =============================================================================
|
||||
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
|
||||
#
|
||||
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
|
||||
# that can `create jobs` in this namespace can create a Job with
|
||||
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
|
||||
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
|
||||
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
|
||||
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
|
||||
# access") while appearing to satisfy it.
|
||||
#
|
||||
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
|
||||
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
|
||||
# the necessary half; that policy is the sufficient half. Neither works alone — do not
|
||||
# grant this Role without that policy reconciled and ENFORCING.
|
||||
#
|
||||
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
rules:
|
||||
# Create the run, and watch it to completion.
|
||||
- apiGroups: [batch]
|
||||
resources: [jobs]
|
||||
verbs: [create, get, list, watch]
|
||||
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
|
||||
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
|
||||
# them from the log, so no execution channel into the runner pod is ever granted.
|
||||
- apiGroups: [""]
|
||||
resources: [pods, pods/log]
|
||||
verbs: [get, list, watch]
|
||||
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
|
||||
# failed run), no secrets, no configmaps, no exec, no create on pods.
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: conformance-trigger
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
@@ -0,0 +1,6 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
|
||||
Install the pinned binary or run via the in-cluster Job (incluster/).
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="115.855688" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,232 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 05:22:26 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1789795346371286043
|
||||
victim node (currently hosts the attached volume): k3s-prod-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-prod-2 labeled
|
||||
node/k3s-prod-2 cordoned
|
||||
cordoned + labelled victim node: k3s-prod-2
|
||||
pod "writer" deleted from chainsaw-advanced-boxer namespace
|
||||
evicted writer off the victim node; volume detached from k3s-prod-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
|
||||
recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-prod-2 uncordoned
|
||||
node/k3s-prod-2 unlabeled
|
||||
restored node to service: k3s-prod-2
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-advanced-boxer namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:24:21 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (115.86s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -5,3 +5,4 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}
|
||||
@@ -0,0 +1,42 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
|
||||
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
|
||||
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
|
||||
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
|
||||
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
|
||||
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
|
||||
#
|
||||
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
|
||||
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
|
||||
# would have skipped this check on the very commit that introduced the defect.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: admission-policies-actually-applied
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
|
||||
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
|
||||
exemption that is genuinely not applied — so a policy cannot be authored, tested,
|
||||
green, and absent from the cluster.
|
||||
spec:
|
||||
steps:
|
||||
- name: every-authored-policy-is-applied-or-documented-ci-only
|
||||
description: >-
|
||||
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
|
||||
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — admission-policies-actually-applied
|
||||
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
|
||||
#
|
||||
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
|
||||
#
|
||||
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
|
||||
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
|
||||
# "kept as reference; THIS file is the buildable render root".
|
||||
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
|
||||
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
|
||||
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
|
||||
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
|
||||
#
|
||||
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
|
||||
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
|
||||
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
|
||||
# `platform` Ready at the new revision — and the policy was never in any applied set. The
|
||||
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
|
||||
# which is precisely the state that identity's own RBAC header says must never exist.
|
||||
#
|
||||
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
|
||||
# DEPLOYED. This check asserts the second thing.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
|
||||
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
|
||||
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
|
||||
# policy that IS applied, so the exemption list cannot rot into a lie either way.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import sys, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
|
||||
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
|
||||
# that silently starts being applied is as much a lie as one that silently stops.
|
||||
CI_ONLY = {
|
||||
"disallow-plaintext-secrets":
|
||||
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
|
||||
"admission because Flux decrypts before the Secret reaches the API server. Its "
|
||||
"in-cluster counterpart is admission/secret-provenance.yaml.",
|
||||
}
|
||||
|
||||
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
|
||||
"platform/kyverno/*.yaml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
|
||||
|
||||
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
|
||||
policies = {}
|
||||
for f in files:
|
||||
try:
|
||||
txt = f.read_text()
|
||||
except Exception:
|
||||
continue
|
||||
if "kind: ClusterPolicy" not in txt:
|
||||
continue
|
||||
for line in txt.splitlines():
|
||||
st = line.strip()
|
||||
if st.startswith("name:") and " name:" in line:
|
||||
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
|
||||
break
|
||||
|
||||
check(len(policies) >= 6,
|
||||
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
|
||||
f"too few means the scan broke, and every assertion below would be vacuous")
|
||||
|
||||
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
|
||||
|
||||
unapplied, wrongly_exempt = [], []
|
||||
for rel, name in sorted(policies.items()):
|
||||
# platform-config references these as ../platform/kyverno/...
|
||||
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
|
||||
if name in CI_ONLY:
|
||||
if referenced:
|
||||
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
|
||||
elif not referenced:
|
||||
unapplied.append(f"{name} ({rel})")
|
||||
|
||||
check(not unapplied,
|
||||
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
|
||||
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
|
||||
|
||||
check(not wrongly_exempt,
|
||||
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
|
||||
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
|
||||
|
||||
for name, reason in CI_ONLY.items():
|
||||
check(bool(reason.strip()),
|
||||
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
|
||||
f"list stops being reviewable")
|
||||
|
||||
check(CHECKS >= 4,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
|
||||
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
|
||||
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
|
||||
f"not that they are correct — `kyverno test` is what proves correctness.")
|
||||
PY
|
||||
@@ -256,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"metricLabelsAllowlist:",
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
|
||||
# The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
|
||||
# end in a literal `]`, asserting the list held exactly this one label — so adding a
|
||||
# second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
|
||||
# while the property it guards stayed perfectly satisfied. A guard that breaks when you
|
||||
# correctly extend the thing it guards teaches people to edit the guard, which is how it
|
||||
# stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
|
||||
@@ -448,9 +454,14 @@ def is_golden_fixture(p: Path) -> bool:
|
||||
return "conformance" in parts and "golden" in parts
|
||||
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
@@ -619,7 +630,7 @@ def yaml_docs(path: Path) -> list[dict]:
|
||||
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
|
||||
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts:
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
|
||||
continue
|
||||
txt = read(p)
|
||||
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
|
||||
|
||||
@@ -179,6 +179,10 @@ UNTESTED_FILES = {
|
||||
# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
|
||||
# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
|
||||
# detectors are no longer unasserted.
|
||||
# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
|
||||
# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
|
||||
# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
|
||||
# had to become an assertion rather than a sentence in a comment.
|
||||
UNTESTED_ALERTS = {
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
||||
@@ -195,7 +199,6 @@ UNTESTED_ALERTS = {
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
||||
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
||||
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
||||
@@ -280,11 +283,16 @@ FETCH_RE = re.compile(
|
||||
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
|
||||
# version, the network and the machine are decided silently at test time, and a missing
|
||||
# toolchain becomes a green run instead of an exit 3.
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
|
||||
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
|
||||
for p in sorted(ROOT.rglob("*.sh")):
|
||||
rel = str(p.relative_to(ROOT))
|
||||
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
|
||||
if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
|
||||
continue
|
||||
try:
|
||||
text = p.read_text()
|
||||
@@ -319,7 +327,7 @@ def tracked_yaml() -> list[Path]:
|
||||
return [ROOT / p for p in out.stdout.split("\0") if p]
|
||||
# No usable index (the gate container mounts a worktree whose .git points outside the
|
||||
# mount). Fall back to a filesystem walk rather than auditing nothing.
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
|
||||
|
||||
|
||||
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
|
||||
@@ -473,7 +481,34 @@ for rel in sorted(alert_files):
|
||||
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
|
||||
f"shrink-only: delete the line so the next missing test is caught")
|
||||
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
|
||||
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
|
||||
# "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
|
||||
# This used to be a bare word match over the whole test file, so an alert named anywhere —
|
||||
# including in a PROSE COMMENT — counted as tested. Hit for real while adding
|
||||
# platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
|
||||
# stays quiet during an egress-steering failure, and that sentence alone was enough to make this
|
||||
# check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
|
||||
# the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
|
||||
# outcome this check exists to prevent, produced BY the check.
|
||||
# Now an alert counts only when a test file names it in an `alertname:` position, with comment
|
||||
# lines stripped first so a commented-out assertion does not count either.
|
||||
# MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
|
||||
# asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
|
||||
# registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
|
||||
# mattered because the registers are shrink-only: had any alert been prose-only, tightening would
|
||||
# have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
|
||||
# it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
|
||||
text_no_comments = "\n".join(
|
||||
l for l in text.split("\n") if not l.lstrip().startswith("#")
|
||||
)
|
||||
# BOTH promtool idioms count, because both are real assertions:
|
||||
# alert_rule_test: - alertname: CNPGPrimariesCoLocated
|
||||
# promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
# Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
|
||||
# platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
|
||||
# form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
|
||||
# already better covered than most.
|
||||
covered = [n for n in names
|
||||
if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
|
||||
covered_total += len(covered)
|
||||
for n in names:
|
||||
if n in covered:
|
||||
|
||||
@@ -126,9 +126,14 @@ def read(p: Path) -> str:
|
||||
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
|
||||
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
|
||||
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
|
||||
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
|
||||
"runbooks", "clusters", "bootstrap", "catalog")
|
||||
|
||||
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
|
||||
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
|
||||
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
|
||||
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
|
||||
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
|
||||
# routes and make (8) demand the operator's dashboard link them.
|
||||
#
|
||||
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
|
||||
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
|
||||
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
|
||||
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
|
||||
# become a real delivery root.
|
||||
NON_SERVING_RECONCILED = ("conformance/incluster",)
|
||||
|
||||
|
||||
def tracked_yaml() -> list[Path]:
|
||||
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
|
||||
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
|
||||
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
|
||||
# a path of them would fail on a file that is correct.
|
||||
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
|
||||
def _reconciled_path(d):
|
||||
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
|
||||
|
||||
|
||||
unscanned = sorted({
|
||||
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
|
||||
for p, d in reconcilers
|
||||
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
|
||||
not in DELIVERY_ROOTS
|
||||
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
|
||||
and _reconciled_path(d) not in NON_SERVING_RECONCILED
|
||||
})
|
||||
|
||||
# The exemption's own guard: an exempted tree must actually serve nothing.
|
||||
exempt_serving = sorted({
|
||||
f"{p.relative_to(ROOT)} ({d.get('kind')})"
|
||||
for p, d in docs_all()
|
||||
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
|
||||
and (d.get("kind") == "HTTPRoute"
|
||||
or (d.get("kind") == "Service"
|
||||
and any("hostname" in str(k).lower()
|
||||
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
|
||||
})
|
||||
check(not exempt_serving,
|
||||
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
|
||||
f"gains a route would hide a service from this check entirely "
|
||||
f"{exempt_serving if exempt_serving else ''}")
|
||||
check(len(reconcilers) >= 10 and not unscanned,
|
||||
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
|
||||
f"reconciled tree this check does not read is a service it cannot require a link for "
|
||||
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
||||
# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
|
||||
# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
|
||||
# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
|
||||
# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
|
||||
# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
|
||||
# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
|
||||
# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
|
||||
# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
|
||||
# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
||||
#
|
||||
# Three things were wrong at once, and none of them was visible from inside the cluster:
|
||||
# * the internal subnet, the Gateway VIP and a complete service inventory were publicly
|
||||
# enumerable by anyone who could guess or brute-force a label;
|
||||
# * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
|
||||
# SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
|
||||
# overrides to reach their own services;
|
||||
# * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
|
||||
# and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
|
||||
# respect.
|
||||
#
|
||||
# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
|
||||
# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
|
||||
# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
|
||||
# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
|
||||
# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
|
||||
# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
|
||||
# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
|
||||
# prose. This test makes the DNS guarantee a gate condition instead.
|
||||
#
|
||||
# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
|
||||
# Service this repo authors,
|
||||
# WHEN each catalog service's rendered routes are resolved in the applied tree (apps/),
|
||||
# THEN (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
|
||||
# annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
|
||||
# uses to skip the source object outright (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch, checked before any hostname is resolved), so the
|
||||
# route cannot become an endpoint for ANY name it declares;
|
||||
# (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
|
||||
# renderer that stamped every route would satisfy (1) perfectly while silently
|
||||
# withdrawing the five public services' records, and `policy: sync` would delete them;
|
||||
# (3) the same holds for the `service` source: an internal service's rendered Service may
|
||||
# not carry an external-dns hostname annotation without the disown annotation beside it.
|
||||
# Two sources are configured, so closing only the route half is half a guarantee;
|
||||
# (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
|
||||
# the emitting function must read `exposure`, must not name any catalog service, and
|
||||
# must not be able to emit external-dns's own `dns-controller` value (the one value that
|
||||
# CLAIMS a route rather than releasing it, inverting the guarantee while looking
|
||||
# plausible). A per-service list is how twelve ServiceMonitors acquired one identical
|
||||
# wrong port name in a single edit;
|
||||
# (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
|
||||
# SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
|
||||
# annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
|
||||
# yield it (the contract's default posture is `internal`, and omitting the field is the
|
||||
# likeliest way a new service arrives). Without this the check would still pass if the
|
||||
# renderer stopped working and someone hand-maintained apps/;
|
||||
# (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
|
||||
# DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
|
||||
# scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
|
||||
# renderer to read, and they are the seven names that REMAIN publicly advertised with an
|
||||
# RFC1918 answer after this change. The register is shrink-only: an entry that stops
|
||||
# matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
|
||||
# quietly grow, and adding a new one is a review event;
|
||||
# AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
|
||||
# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
|
||||
# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
|
||||
# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
|
||||
# Same reasoning as its whole-repo siblings.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
|
||||
# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
|
||||
# and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
|
||||
# - That the record is actually GONE from Cloudflare. That is a live read against an external
|
||||
# authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
|
||||
# the positive direction (a public name DOES appear). The negative direction was proven
|
||||
# MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
|
||||
# carrying the annotation produced no record at all, and adding the annotation to a route that
|
||||
# already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
|
||||
# entry within one ~60s reconcile — but a manual proof is not a gate.
|
||||
# - That an `internal` name is unreachable from the internet. No DNS record is necessary for
|
||||
# reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
|
||||
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
|
||||
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
|
||||
# each publishes so the claim is reviewable, which is all a static check can offer.
|
||||
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
|
||||
# rather than a routable address; that is an open operator decision recorded in
|
||||
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: internal-services-publish-no-public-dns
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-02: "true"
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
|
||||
# HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
|
||||
# Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
|
||||
# ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
|
||||
# at all, so a requirement-scoped selection would skip this check on precisely the changes
|
||||
# that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
A service the operator declared `internal` cannot produce a public DNS record: the
|
||||
guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
|
||||
in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
|
||||
and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
|
||||
register so the unadjudicated set cannot grow silently.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-no-internal-service-can-publish-a-public-dns-record
|
||||
description: >-
|
||||
Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
|
||||
disown annotation on every internal one and on no public one, extend the same rule to the
|
||||
`service` source, assert catalog/render.py derives the annotation from `exposure` rather
|
||||
than from a service list, prove the shipped renderer on internal/public/default-posture
|
||||
fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+458
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
|
||||
# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
|
||||
#
|
||||
# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
|
||||
# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
|
||||
# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
|
||||
# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
|
||||
# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
|
||||
# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
|
||||
# what keeps it derived, present, and scoped.
|
||||
#
|
||||
# ASSERTIONS
|
||||
# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
|
||||
# (1) every rendered route of an `internal` service carries the disown annotation;
|
||||
# (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
|
||||
# satisfy (1) and silently delete the five public records under policy: sync;
|
||||
# (3) the `service` source half: an internal service's Service may not carry an external-dns
|
||||
# hostname annotation without the disown annotation beside it;
|
||||
# (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
|
||||
# field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
|
||||
# (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
|
||||
# public, and `exposure` OMITTED — the contract's default posture is internal);
|
||||
# (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
|
||||
# KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
|
||||
# (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
|
||||
# ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
|
||||
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
|
||||
# - That the record is gone from Cloudflare. Live read; belongs beside
|
||||
# conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
|
||||
# direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
|
||||
# annotation produced no record; adding it to a route that had one produced action=DELETE for
|
||||
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
|
||||
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
|
||||
# This check owns the NAME axis only.
|
||||
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
|
||||
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
|
||||
# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
|
||||
# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
|
||||
if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
|
||||
echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
|
||||
|
||||
PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
|
||||
line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
|
||||
structures, and a line-based reader of them is precisely the approximate parser that silently
|
||||
stops matching — the failure class this check exists to prevent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
|
||||
# ControllerKey = annotationPrefix + "controller" -> the key below
|
||||
# ControllerValue = "dns-controller" -> the value that CLAIMS an object
|
||||
# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
|
||||
# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
|
||||
CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
CONTROLLER_CLAIMED = "dns-controller"
|
||||
HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
def docs_of(p: Path):
|
||||
"""Every YAML document in a file, tolerating the non-manifest files in the tree."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
|
||||
#
|
||||
# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
|
||||
# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
|
||||
# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
|
||||
# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
|
||||
# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
|
||||
# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
|
||||
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
|
||||
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
|
||||
#
|
||||
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
|
||||
# the honest state after this change, and it is deliberate: converting them is an operator
|
||||
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
|
||||
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
|
||||
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
|
||||
#
|
||||
# Keyed by "<relpath>:<namespace>/<name>".
|
||||
# =============================================================================================
|
||||
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
|
||||
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
|
||||
"platform component, not a catalog service; it must be reachable by every browser that "
|
||||
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
|
||||
"whether the five public services are — an operator decision, not a renderer one.",
|
||||
"platform/keycloak/httproute.yaml:keycloak/keycloak":
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
|
||||
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
|
||||
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
|
||||
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
|
||||
"is meant to be public either way; it is listed because the PUBLISHING object is the "
|
||||
"hand-authored platform route, which carries no exposure field.",
|
||||
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
|
||||
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
|
||||
"certainly should not be publicly resolvable, but it is not a catalog service and "
|
||||
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
|
||||
"internal services (Pi-hole must answer first).",
|
||||
"platform/headlamp/httproute.yaml:headlamp/headlamp":
|
||||
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
|
||||
"Keycloak OIDC; same adjudication as grafana.",
|
||||
"platform/homepage/httproute.yaml:homepage/homepage":
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
|
||||
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
|
||||
"declaration.",
|
||||
"runbooks/serving/httproute.yaml:runbooks/runbooks":
|
||||
"declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
|
||||
"contributes to the apex record rather than a name of its own (the apex TXT registry "
|
||||
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
|
||||
"header for why gating runbooks on Keycloak is a loop with no exit.",
|
||||
"platform/crafty/httproute.yaml:crafty/crafty":
|
||||
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
|
||||
"oauth2-proxy sidecar). Platform component, no exposure field.",
|
||||
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
|
||||
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
|
||||
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
|
||||
"Gateway cannot express. A game server is the one case where a public name may be "
|
||||
"genuinely wanted; the address it answers with is still RFC1918.",
|
||||
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
|
||||
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
|
||||
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
|
||||
"vm-demo, which declares no ingressRoutes and publishes nothing.",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
|
||||
# =============================================================================================
|
||||
CATALOG = ROOT / "catalog"
|
||||
sds: dict[str, dict] = {}
|
||||
for p in sorted(CATALOG.glob("*.yaml")):
|
||||
for d in docs_of(p):
|
||||
if "name" in d and "source" in d:
|
||||
sds[d["name"]] = d
|
||||
|
||||
check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
|
||||
|
||||
# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
|
||||
# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
|
||||
# renderer: a new service that omits exposure would go unaudited.
|
||||
exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
|
||||
# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
|
||||
# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
|
||||
expected_routes: dict[tuple[str, str], str] = {}
|
||||
for n, sd in sds.items():
|
||||
for r in sd.get("ingressRoutes") or []:
|
||||
expected_routes[(n, f"{n}-{r['port']}")] = n
|
||||
|
||||
internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "internal"})
|
||||
public_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "public"})
|
||||
check(len(internal_with_routes) > 0,
|
||||
"inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
|
||||
"this check is empty, which means either the catalog changed shape or the parse is wrong")
|
||||
check(len(public_with_routes) > 0,
|
||||
"inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
|
||||
"vacuous, so the parse is presumed wrong rather than the repo")
|
||||
|
||||
SCAN_DIRS = ["apps", "platform", "runbooks"]
|
||||
routes: list[tuple[str, dict]] = [] # (relpath, HTTPRoute doc)
|
||||
dns_services: list[tuple[str, dict]] = [] # (relpath, Service doc carrying an external-dns ann)
|
||||
for sub in SCAN_DIRS:
|
||||
base = ROOT / sub
|
||||
if not base.is_dir():
|
||||
continue
|
||||
for p in sorted(base.rglob("*.yaml")):
|
||||
rel = p.relative_to(ROOT).as_posix()
|
||||
for d in docs_of(p):
|
||||
kind = d.get("kind")
|
||||
md = d.get("metadata") or {}
|
||||
ann = md.get("annotations") or {}
|
||||
if kind == "HTTPRoute":
|
||||
routes.append((rel, d))
|
||||
elif kind == "Service" and any(
|
||||
a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
|
||||
):
|
||||
dns_services.append((rel, d))
|
||||
|
||||
check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
|
||||
"contract broken (the tree is known to author several)")
|
||||
|
||||
|
||||
def ident(d: dict) -> tuple[str, str]:
|
||||
md = d.get("metadata") or {}
|
||||
return (md.get("namespace") or "", md.get("name") or "")
|
||||
|
||||
|
||||
def annotations_of(d: dict) -> dict:
|
||||
return (d.get("metadata") or {}).get("annotations") or {}
|
||||
|
||||
|
||||
def is_disowned(d: dict) -> bool:
|
||||
"""True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
|
||||
and its value is not external-dns's own. Absent == published (the default), which is the whole
|
||||
point — the annotation is the only thing standing between a route and the public zone."""
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
return v is not None and v != CONTROLLER_CLAIMED
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# (1) + (2) every internal service's rendered route is disowned; no public one is.
|
||||
# =============================================================================================
|
||||
by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
|
||||
for rel, d in routes:
|
||||
by_ident.setdefault(ident(d), []).append((rel, d))
|
||||
|
||||
for (ns, rname), svc in sorted(expected_routes.items()):
|
||||
found = by_ident.get((ns, rname), [])
|
||||
check(len(found) >= 1,
|
||||
f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
|
||||
f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
|
||||
f"(ci/render-golden.sh owns the byte-level half of this)")
|
||||
for rel, d in found:
|
||||
if exposure_of[svc] == "internal":
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
check(
|
||||
v is not None,
|
||||
f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
|
||||
f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
|
||||
f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
|
||||
f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
|
||||
)
|
||||
check(
|
||||
v != CONTROLLER_CLAIMED,
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
|
||||
f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
|
||||
f"rather than releasing it. It reads like the guard and is its exact inverse.",
|
||||
)
|
||||
else:
|
||||
check(
|
||||
CONTROLLER_KEY not in annotations_of(d),
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
|
||||
f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
|
||||
f"`policy: sync` that DELETES its live public record — the five public services "
|
||||
f"must keep publishing.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (3) the `service` source half — an internal service's Service may not publish either.
|
||||
# =============================================================================================
|
||||
for rel, d in dns_services:
|
||||
ns, name = ident(d)
|
||||
svc = ns if ns in sds else None
|
||||
if svc and exposure_of[svc] == "internal":
|
||||
check(
|
||||
is_disowned(d),
|
||||
f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
|
||||
f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
|
||||
f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
|
||||
f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
|
||||
f"half is half a guarantee.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
|
||||
# =============================================================================================
|
||||
render_py = ROOT / "catalog" / "render.py"
|
||||
src = read(render_py)
|
||||
check(bool(src), "catalog/render.py unreadable")
|
||||
|
||||
emitter = re.search(
|
||||
r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
|
||||
)
|
||||
check(emitter is not None,
|
||||
"catalog/render.py must define `_externaldns_route_annotations` — the single place the "
|
||||
"ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
|
||||
"read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
|
||||
if emitter:
|
||||
body = emitter.group(0)
|
||||
# The derivation must read `exposure`. A body that does not mention it cannot be deriving
|
||||
# anything from the operator's declaration.
|
||||
check('"exposure"' in body or "'exposure'" in body,
|
||||
"`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
|
||||
"that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
|
||||
# and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
|
||||
# acquired one identical wrong port name in a single edit. Only the executable statements are
|
||||
# inspected — the docstring legitimately names the affected services as evidence.
|
||||
code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
|
||||
named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
|
||||
check(not named,
|
||||
f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
|
||||
f"annotation must be derived from `exposure` alone; a per-service list silently omits "
|
||||
f"the next service anyone adds.")
|
||||
check(CONTROLLER_CLAIMED not in code,
|
||||
f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
|
||||
f"external-dns's own value CLAIMS the object instead of releasing it")
|
||||
|
||||
# =============================================================================================
|
||||
# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
|
||||
# "internal is annotated" alone would also pass if the renderer annotated everything, and
|
||||
# because the contract's DEFAULT posture is internal — the case a new service hits by
|
||||
# omitting the field entirely, and the case a per-service list would miss.
|
||||
# =============================================================================================
|
||||
FIXTURES = [
|
||||
("fixture-internal", {"exposure": "internal"}, True,
|
||||
"an `internal` fixture must render the disown annotation"),
|
||||
("fixture-public", {"exposure": "public"}, False,
|
||||
"a `public` fixture must render NO disown annotation"),
|
||||
("fixture-default", {}, True,
|
||||
"a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
|
||||
"must also render the disown annotation"),
|
||||
]
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
for fname, extra, want, msg in FIXTURES:
|
||||
sd = {
|
||||
"name": fname,
|
||||
"kind": "container",
|
||||
"source": {"image": "example/app:1.0.0"},
|
||||
"egress": "internet",
|
||||
"dependencies": [],
|
||||
"ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
|
||||
# `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
|
||||
# keeping the fixture about DNS scoping and nothing else. A public service may not use
|
||||
# auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
|
||||
"auth": {"mode": "native"},
|
||||
"update": {"channel": "stable", "cadence": "0 4 * * 0",
|
||||
"autoRollbackOnFailedHealth": True},
|
||||
"observability": {"metrics": True, "slis": ["up"]},
|
||||
"backup": {"policy": "reacquirable"},
|
||||
}
|
||||
sd.update(extra)
|
||||
f = Path(td) / f"{fname}.yaml"
|
||||
f.write_text(yaml.safe_dump(sd, sort_keys=False))
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(render_py), str(f), "--no-image-automation"],
|
||||
capture_output=True, text=True, cwd=str(ROOT),
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
check(False, f"fixture {fname}: the shipped renderer failed: "
|
||||
f"{proc.stderr.strip()[:400]}")
|
||||
continue
|
||||
rendered = [d for d in yaml.safe_load_all(proc.stdout)
|
||||
if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
|
||||
check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
|
||||
if rendered:
|
||||
got = is_disowned(rendered[0])
|
||||
check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
|
||||
f"{annotations_of(rendered[0])})")
|
||||
|
||||
# =============================================================================================
|
||||
# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
|
||||
# declaration still names a real object.
|
||||
# =============================================================================================
|
||||
catalog_rendered = set(expected_routes)
|
||||
seen_register_keys: set[str] = set()
|
||||
for rel, d in routes + dns_services:
|
||||
ns, name = ident(d)
|
||||
if (ns, name) in catalog_rendered:
|
||||
continue
|
||||
key = f"{rel}:{ns}/{name}"
|
||||
seen_register_keys.add(key)
|
||||
check(
|
||||
key in KNOWN_NON_CATALOG_PUBLISHERS,
|
||||
f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
|
||||
f"`exposure` field this check can adjudicate, and it can put a name into the public "
|
||||
f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
|
||||
f"it is out of scope — or render it from a ServiceDefinition instead.",
|
||||
)
|
||||
stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
|
||||
check(not stale,
|
||||
f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
|
||||
f"name no object in the tree any more. Delete them — a register that keeps entries for "
|
||||
f"deleted objects stops being evidence of anything.")
|
||||
|
||||
# =============================================================================================
|
||||
# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
|
||||
# always selects this test) and NO stage is hand-written into ci/l0.sh (the
|
||||
# whole-repo-invariants-wired meta-test fails the gate if one is).
|
||||
# =============================================================================================
|
||||
TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
|
||||
test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
|
||||
check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
|
||||
check(
|
||||
re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
|
||||
is not None,
|
||||
"chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
|
||||
"mapping line — that declaration is the only thing that unions this check into every gate "
|
||||
"run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
|
||||
)
|
||||
l0_sh = read(ROOT / "ci/l0.sh")
|
||||
check(
|
||||
"internal-services-publish-no-public-dns" not in l0_sh,
|
||||
"ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
|
||||
"and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
print(f"check FAIL: {f}", file=sys.stderr)
|
||||
print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: {CHECKS} assertions passed "
|
||||
f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
|
||||
f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
|
||||
f"{len(dns_services)} external-dns-annotated Services inventoried; "
|
||||
f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
|
||||
PY
|
||||
@@ -238,7 +238,7 @@ def bail(msg):
|
||||
|
||||
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
|
||||
SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
|
||||
SKIP_DIRS = {".git", "_artifacts", "node_modules", ".venv"}
|
||||
SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
|
||||
|
||||
# ------------------------------------------------------------------ discover the guard FILE
|
||||
# Hardcoding a path is what let a file that never landed disable this whole check.
|
||||
|
||||
@@ -154,8 +154,13 @@ services: list[tuple] = []
|
||||
monitors: list[tuple] = []
|
||||
unparseable: list[str] = []
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts or is_test_fixture(p) or ".sops." in p.name:
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
|
||||
continue
|
||||
txt = read(p)
|
||||
if not txt:
|
||||
|
||||
@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
|
||||
fi
|
||||
|
||||
grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH"
|
||||
chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
"conformance/lib/common.sh no longer resolves the scope annotation — no test can be
|
||||
selected BY declaring a scope, so every whole-repo invariant is back to being
|
||||
selectable away by the requirement filter"
|
||||
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
|
||||
chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
"conformance/run.sh does not union scope-declared tests (--include-scope /
|
||||
conf_dirs_for_scopes missing) — the annotation would be inert decoration"
|
||||
|
||||
grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
|
||||
chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
"ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
|
||||
purely requirement-scoped run, and every whole-repo invariant is filtered out of
|
||||
exactly the change that breaks it"
|
||||
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
|
||||
CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
|
||||
)
|
||||
MIN_DECLARED="${MIN_DECLARED:-5}"
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
|
||||
chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
|
||||
chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
"only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
|
||||
annotation was stripped from the whole-repo checks, or the resolver stopped matching
|
||||
it. Every assertion below would then pass while checking nothing"
|
||||
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
rel="${d#"$ROOT"/}"
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
|
||||
chk $? "${rel}: declares the scope AND carries an executable offline check" \
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
|
||||
chk $arc "${rel}: declares the scope AND carries an executable offline check" \
|
||||
"${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
|
||||
unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
|
||||
permanent not-green, not a guard"
|
||||
@@ -91,25 +91,58 @@ BOGUS_REQ="r-nosuchreq-99"
|
||||
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
|
||||
sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
|
||||
|
||||
[ -z "$sel_without" ]
|
||||
chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
[ -z "$sel_without" ] && arc=0 || arc=$?
|
||||
chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
"the control case is broken: --requirements ${BOGUS_REQ} still selected tests
|
||||
($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
|
||||
about --include-scope"
|
||||
|
||||
# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
|
||||
# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
|
||||
# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
|
||||
# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
|
||||
# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
|
||||
# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
|
||||
missing=""
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
|
||||
printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ -z "$missing" ]
|
||||
chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
[ -z "$missing" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
"the scope union does not actually select:${missing}. This is the blind spot itself: a
|
||||
change that attributes no requirement would not run these checks"
|
||||
|
||||
# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
|
||||
# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
|
||||
# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
|
||||
# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
|
||||
# three rendering goldens stale.
|
||||
sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
|
||||
--print-selection 2>/dev/null || true)"
|
||||
missing_l1=""
|
||||
declared_l1=0
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
|
||||
declared_l1=$((declared_l1+1))
|
||||
printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
|
||||
"scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
|
||||
${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
|
||||
when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
|
||||
|
||||
# And the L1 union must not leak the other way either.
|
||||
strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
|
||||
[ -z "$strays_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
|
||||
"--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
|
||||
|
||||
# The union must not smuggle in another layer's tests when --layer bounds the run.
|
||||
strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
|
||||
[ -z "$strays" ]
|
||||
chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
[ -z "$strays" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
"--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
|
||||
|
||||
# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
|
||||
@@ -143,23 +176,23 @@ run_offline() { # run_offline <dir...> -> echoes rc
|
||||
rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
|
||||
rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
|
||||
|
||||
[ "$rc_declared" = 3 ]
|
||||
chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
[ "$rc_declared" = 3 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
"a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
|
||||
whole-repo invariant reporting 'nothing to audit' means the tree it audits has
|
||||
VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
|
||||
rc=3. Exit 0 here turns that back into a silent green"
|
||||
|
||||
[ "$rc_undeclared" = 0 ]
|
||||
chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
"a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
|
||||
rule leaked onto ordinary scaffolds, which would make every unfinished test block"
|
||||
|
||||
# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
|
||||
l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
|
||||
printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
|
||||
chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
"ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
|
||||
check that SKIPPED would be reported as a PASS — the exact false green the deleted
|
||||
stages were written to prevent"
|
||||
@@ -193,8 +226,8 @@ done
|
||||
|
||||
# ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
|
||||
# Carried over from the three deleted stages' own checks — asserted once, here.
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
|
||||
chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
"ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
|
||||
would hold while a red L0 stage merged anyway"
|
||||
|
||||
|
||||
@@ -23,6 +23,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-PLAT-02"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
|
||||
spec:
|
||||
|
||||
@@ -196,6 +196,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -27,6 +27,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
Rendering any service's namespace yields a default-deny baseline policy.
|
||||
spec:
|
||||
|
||||
@@ -32,6 +32,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
egress: vpn renders the egress-gateway + kill-switch path; internet does not.
|
||||
spec:
|
||||
|
||||
+1
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: chore/decisions-log-conflict-free
|
||||
title: The handoff log was serialising every parallel branch — it is a directory now
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`DECISIONS-LOG.md` is frozen and the handoff log is now a directory, `decisions/`, with one
|
||||
file per entry (`YYYY-MM-DD-<slug>.md`). Concurrent branches write disjoint paths, so there
|
||||
is nothing for git to merge.
|
||||
|
||||
The problem was not any particular change. CLAUDE.md §7 told every agent to append a handoff,
|
||||
and appending always landed on the *same last hunk* of one 5,228-line file. Two branches that
|
||||
both did it could not auto-merge; N branches cost N-1 rebases, and each rebase invalidated the
|
||||
gate run before it. The agents work in isolated worktrees precisely so they don't block each
|
||||
other, and this one file undid that.
|
||||
|
||||
- `decisions/README.md` — the convention, the template, and how to read the log. No index file
|
||||
and no sequence number, deliberately: an index is a shared mutable file (the thing that just
|
||||
failed) and a counter has two agents racing for the same number. The directory listing is the
|
||||
index.
|
||||
- `DECISIONS-LOG.md` — a "FROZEN ARCHIVE" header prepended; content otherwise untouched. Its
|
||||
SHA-256 is pinned in `decisions/ARCHIVE.sha256`.
|
||||
- `ci/decisions-log.sh` — a new gate stage. Asserts the archive matches its pin, and that every
|
||||
entry is named `YYYY-MM-DD-<slug>.md` with a real date, carries `date`/`branch`/`title`
|
||||
frontmatter agreeing with its filename, and has all three of §7's sections with a real body
|
||||
under each. Exit 0 or 1 only; it has no skip path by construction, because it reads only files
|
||||
in the tree under test and needs no tool and no cluster.
|
||||
- `ci/gate.sh` — runs the stage; both `fail` and (impossible) `skip` set `BLOCK`. There is no
|
||||
`REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard.
|
||||
- `ci/tests/decisions-log.test.sh` — 22 assertions, new.
|
||||
- `ci/tests/gate-decision.test.sh` — 3 new assertions for the stage's wiring, **plus a
|
||||
pre-existing bug fixed**: its stub list never included `gitea-settings`, so that stage's
|
||||
script was absent from the stub CI dir, `run_stage` recorded exit 127 as a FAILED stage in
|
||||
every case, and 15 of 47 assertions failed on `main`. The suite was measuring its own missing
|
||||
stub rather than the decision logic.
|
||||
- `CLAUDE.md` §7 rewritten; §1's "stale runbook wording" caveat extended to the log.
|
||||
- `ci/Makefile`, `ci/README.md`, `.gitea/workflows/README.md` — the stage named where stages are
|
||||
named.
|
||||
|
||||
A guard rather than a paragraph, because the old habit is written into the runbooks and into
|
||||
every agent that has read the log, and getting it wrong does not fail: it produces a clean-looking
|
||||
commit that re-creates the conflict on *other people's* branches, later, misattributed.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured against the live Gitea (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`,
|
||||
2026-09-29), not inferred from the tree:
|
||||
|
||||
#22 fix/crafty-oauth2-proxy-signin mergeable=False
|
||||
#20 feat/conformance-job-template mergeable=False
|
||||
#19 feat/conformance-trigger-kubeconfig mergeable=False
|
||||
#18 feat/conformance-runner-image mergeable=False
|
||||
#17 feat/ci-is-the-approver mergeable=True
|
||||
|
||||
`git merge-tree --write-tree origin/main origin/<branch>` for each named
|
||||
`CONFLICT (content): DECISIONS-LOG.md` and **no other conflicting path** in all four. #17 is
|
||||
clean and touched no log entry. So the log's storage format was the whole of the blockage.
|
||||
|
||||
Tests: `ci/tests/decisions-log.test.sh` **PASS (22 assertions)**, including the one that
|
||||
matters — an append to `DECISIONS-LOG.md` exits non-zero and the message names both the frozen
|
||||
archive and the file to write instead. `ci/tests/gate-decision.test.sh` **PASS (50 assertions)**,
|
||||
up from **FAIL (15 failed, 32 passed)** on `main` before the stub fix.
|
||||
|
||||
Gate: see the run quoted in the PR description.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing was applied and nothing was merged.** No `kubectl`, no `flux reconcile`. This change
|
||||
touches only CI and documentation; it has no cluster-visible effect at all.
|
||||
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), as always. Irrelevant to this
|
||||
change's content but stated so the gate result is not over-read.
|
||||
- **I have not watched a Gitea PR check run this stage.** The stage is proven by its own unit
|
||||
test and by the gate locally; that the `incluster-dind` runner executes it is unobserved here.
|
||||
A skipped Gitea job still reports `success`, so read the job log, not the tick.
|
||||
- **The four hung PRs are not yet fixed by this commit.** They still carry their
|
||||
`DECISIONS-LOG.md` appends and will still conflict until each is rebased onto this change with
|
||||
its append converted to a `decisions/` entry. That is follow-up work, in flight separately.
|
||||
- **`runbooks/` and older entries still say "append to DECISIONS-LOG.md"** in places. I did not
|
||||
sweep them. The gate now rejects the action, so the stale wording misleads but cannot cause
|
||||
the damage; it joins the existing tracked work on stale `git push origin main` wording.
|
||||
@@ -0,0 +1 @@
|
||||
b16e8cb7ea12720bc7c3a77a496211e3204534c5e6600ba748fb9ab14fa21c26 DECISIONS-LOG.md
|
||||
@@ -0,0 +1,101 @@
|
||||
# `decisions/` — the handoff log, one file per entry
|
||||
|
||||
This directory **is** the decision log. `../DECISIONS-LOG.md` is the frozen archive of
|
||||
everything written before 2026-09-29 and is never edited again (see "The archive is
|
||||
frozen" below).
|
||||
|
||||
## Why this exists
|
||||
|
||||
`DECISIONS-LOG.md` was append-only in a single file, and every agent appended to its
|
||||
**tail**. Two branches that both append hit the same last hunk, so git cannot
|
||||
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
|
||||
the fourth. Measured on 2026-09-29: **four of the five open PRs were unmergeable, and
|
||||
`DECISIONS-LOG.md` was the only conflicting file in all four**:
|
||||
|
||||
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
|
||||
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
|
||||
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
|
||||
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
|
||||
#17 feat/ci-is-the-approver clean (it touched no log entry)
|
||||
|
||||
Nothing was wrong with any of those changes. The log's *storage format* serialised work
|
||||
that was otherwise parallel, and it got worse with every concurrent branch: N branches
|
||||
produce N-1 rebases, each of which invalidates any gate run that preceded it.
|
||||
|
||||
The fix is the obvious one for concurrent appenders: **stop sharing a file**. Each
|
||||
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
|
||||
nothing for git to merge and nothing to conflict. This scales to any number of parallel
|
||||
branches with zero coordination — which is the point, since the agents are deliberately
|
||||
isolated from each other in separate worktrees and cannot coordinate.
|
||||
|
||||
There is deliberately **no index file** and no counter. An index would be a shared
|
||||
mutable file — exactly the thing that just failed — and a sequence number would have
|
||||
two agents racing for `0043`. The directory listing is the index.
|
||||
|
||||
## Writing an entry
|
||||
|
||||
One file per handoff, named:
|
||||
|
||||
decisions/YYYY-MM-DD-<slug>.md
|
||||
|
||||
`<slug>` is lowercase `[a-z0-9-]`. **Use your branch name's last segment** — it is
|
||||
already unique per unit of work, which is precisely the property the filename needs.
|
||||
A branch `fix/crafty-oauth2-proxy-signin` handed off on 2026-09-29 writes:
|
||||
|
||||
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
|
||||
|
||||
The file is YAML frontmatter followed by three required sections:
|
||||
|
||||
```markdown
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: fix/crafty-oauth2-proxy-signin
|
||||
title: The Keycloak gate works — Crafty's own login was the wall
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Prose. What you actually did, and why.
|
||||
|
||||
## What I verified
|
||||
|
||||
The measurements, with their numbers. Quote the field you read, not the prose you
|
||||
believed. Name the gate result and its exit code.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
|
||||
this repo has been burned by the difference more than once.
|
||||
```
|
||||
|
||||
All three headings are **required and enforced** (`ci/decisions-log.sh`). They are
|
||||
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
|
||||
degrading into "done ✅".
|
||||
|
||||
## Reading the log
|
||||
|
||||
The whole log, oldest first:
|
||||
|
||||
ls decisions/[0-9]*.md | sort
|
||||
|
||||
The current PM handoff — the newest entry. Ties within a day are broken by what landed
|
||||
on `main` last, which is the honest answer, so ask git rather than the filename:
|
||||
|
||||
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
|
||||
| grep '^decisions/' | head -1
|
||||
|
||||
For a quick skim of who said what:
|
||||
|
||||
grep -H '^title:' decisions/[0-9]*.md | sort
|
||||
|
||||
## The archive is frozen
|
||||
|
||||
`DECISIONS-LOG.md` is **read-only history**. Its SHA-256 is pinned in
|
||||
`decisions/ARCHIVE.sha256` and `ci/decisions-log.sh` fails the merge gate if the file
|
||||
changes. That is not ceremony: without it, the next agent to read `runbooks/` or an old
|
||||
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
|
||||
else, silently, one PR at a time.
|
||||
|
||||
If the archive genuinely must change (a leaked credential to redact, say), that is a
|
||||
deliberate, reviewed act: change it, run `sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256`,
|
||||
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.
|
||||
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
|
||||
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
|
||||
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
|
||||
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
|
||||
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
|
||||
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
|
||||
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
|
||||
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
|
||||
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
|
||||
> 2026-09-29, lie about it.
|
||||
>
|
||||
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
|
||||
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
|
||||
> not enable L2/L3, it only changes which way this stage fails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
# infra/gitea — platform settings that are NOT GitOps-managed
|
||||
|
||||
Gitea's own repo settings (branch protection, required status checks, approvals) live
|
||||
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
|
||||
and invisible to the gate** — exactly how the state below went unnoticed.
|
||||
|
||||
## The gap this closes
|
||||
|
||||
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
|
||||
(2026-08-31) asserting the opposite:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
|
||||
|
||||
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
|
||||
rule; this directory is how the rule gets enforced by the machine.
|
||||
|
||||
## `branch-protection.sh`
|
||||
|
||||
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
|
||||
do not trust the API response**, since Gitea may let site admins bypass protection.
|
||||
|
||||
./infra/gitea/branch-protection.sh
|
||||
|
||||
It authenticates with the credential `git` already has stored for this host, and never
|
||||
prints the secret.
|
||||
|
||||
## `runner-token-rotate.sh`
|
||||
|
||||
Mints a fresh instance-level registration token from the Gitea API and writes it into
|
||||
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
|
||||
never printed.**
|
||||
|
||||
./infra/gitea/runner-token-rotate.sh
|
||||
|
||||
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
|
||||
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
|
||||
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
|
||||
and polls with an identity Gitea does not recognise — so it never appears in the runner
|
||||
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
|
||||
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
||||
commands.
|
||||
|
||||
## Sequence — DONE 2026-09-29
|
||||
|
||||
All four steps are complete and each was verified, not assumed:
|
||||
|
||||
| Step | State |
|
||||
|---|---|
|
||||
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
||||
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
||||
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
||||
|
||||
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
||||
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
||||
account agents authenticate as.
|
||||
|
||||
### 5. Block the admin override — the step the first pass MISSED
|
||||
|
||||
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
|
||||
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
|
||||
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
|
||||
like the gate were overridable by precisely the account that merges.
|
||||
|
||||
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
|
||||
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
|
||||
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
|
||||
(`/branch_protections/main`) and enumerate its keys.
|
||||
|
||||
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
|
||||
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
|
||||
> means "key absent". Enumerate with `to_entries` and act on what is really there.
|
||||
|
||||
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
|
||||
proven when a green PR is REFUSED a merge until approved, and not before.
|
||||
|
||||
### `require-gate.sh` / `verify-then-enforce.sh`
|
||||
|
||||
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
|
||||
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
|
||||
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
|
||||
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
|
||||
|
||||
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
||||
|
||||
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
||||
|
||||
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
||||
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
||||
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
||||
database.
|
||||
|
||||
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
||||
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
||||
|
||||
Behaviour, proven by test rather than assumed:
|
||||
|
||||
| Situation | Exit | Effect |
|
||||
|---|---|---|
|
||||
| live matches the declaration | 0 | green |
|
||||
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
||||
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
||||
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
||||
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
||||
`gate.yml` so a future missing token blocks instead of skipping.
|
||||
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
conformance gate can read. Nothing detects drift from what this directory declares. That is
|
||||
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
|
||||
gone for a month while the log recorded both as working, and **no check existed that could
|
||||
have caught it.**
|
||||
|
||||
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
||||
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
||||
root cause of the 2026-09-29 session.
|
||||
|
||||
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. That is the remaining step.
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
### Original ordering notes
|
||||
|
||||
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
||||
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
||||
skip the confirmation: the whole reason this went unnoticed is that a runner was
|
||||
*recorded* as online without anyone checking.
|
||||
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
||||
3. **Only then set `enable_status_check: true`** with that exact context.
|
||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||
|
||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
||||
> reported.
|
||||
|
||||
### Measured, once the runner came up (2026-09-29)
|
||||
|
||||
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
||||
|
||||
| Context | Result | Time |
|
||||
|---|---|---|
|
||||
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
||||
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
||||
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
||||
|
||||
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
||||
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
||||
does not hang a status check: Gitea reports it as **`success`** with the description
|
||||
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
||||
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
||||
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
||||
exactly the fail-open shape this repo keeps finding.
|
||||
|
||||
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
||||
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
||||
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
||||
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stop repository admins from overriding the merge requirements on `main`.
|
||||
#
|
||||
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
|
||||
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
|
||||
# required_approvals and the required status check were both overridable by the one account
|
||||
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
|
||||
# `required_approvals: 1`.
|
||||
#
|
||||
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
|
||||
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
|
||||
# real keys (`to_entries`) rather than probing names you assume exist.
|
||||
#
|
||||
# Recoverable: an owner can PATCH this back to false if it gets in the way.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
|
||||
|
||||
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
|
||||
api -X PATCH -H 'Content-Type: application/json' \
|
||||
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
|
||||
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
|
||||
echo "==> after: block_admin_merge_override = $AFTER"
|
||||
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
|
||||
|
||||
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
|
||||
cat <<'NOTE'
|
||||
|
||||
NOW PROVE IT -- the setting is not the evidence. On the next PR:
|
||||
* let the gate go green, then try to merge WITHOUT approving it.
|
||||
* Gitea must REFUSE. If it merges, this still is not doing its job.
|
||||
Do not record approvals as enforced until you have seen a merge refused.
|
||||
NOTE
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# Enable branch protection on main for home-network-k3s-impl.
|
||||
#
|
||||
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
|
||||
#
|
||||
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
|
||||
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
|
||||
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
|
||||
# Turn it on only after a runner is live and you have seen the gate actually report on
|
||||
# a real PR, using the context name CI itself produced.
|
||||
#
|
||||
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
|
||||
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
|
||||
# Add the collaborator first, then raise this to 1.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
|
||||
echo "==> creating branch protection on 'main'"
|
||||
jq -n '{
|
||||
branch_name: "main",
|
||||
enable_push: false,
|
||||
enable_status_check: false,
|
||||
status_check_contexts: [],
|
||||
required_approvals: 0,
|
||||
block_on_rejected_reviews: true,
|
||||
block_on_outdated_branch: false,
|
||||
dismiss_stale_approvals: true
|
||||
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
-X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
|
||||
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
|
||||
|
||||
echo "==> verifying (this is the part that matters)"
|
||||
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Now prove it actually blocks you -- do not take the API's word for it. From a clean
|
||||
checkout on main, with nothing to lose:
|
||||
|
||||
git commit --allow-empty -m 'protection probe'
|
||||
git push origin main # EXPECT: rejected by the remote
|
||||
git reset --hard HEAD~1 # tidy up either way
|
||||
|
||||
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
|
||||
yet doing what you want.
|
||||
NOTE
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
"enable_push": false,
|
||||
"enable_push_whitelist": false,
|
||||
"enable_merge_whitelist": false,
|
||||
"enable_status_check": true,
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
"dismiss_stale_approvals": true
|
||||
}
|
||||
}
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
|
||||
#
|
||||
# RUN THIS LAST. Prerequisites, in order:
|
||||
# a) PR #7 (dind native sidecar) is MERGED, and
|
||||
# b) you have SEEN a real workflow run go green on the converted pod.
|
||||
# The runner Deployment is the merge gate for the whole repo. Turning on a required check
|
||||
# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
|
||||
# an owner can set enable_status_check back to false. But do not rely on that.)
|
||||
#
|
||||
# Requires exactly ONE context: `gate / gate (pull_request)`.
|
||||
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
|
||||
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
|
||||
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
|
||||
# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
|
||||
# lets repo admins override merge requirements without it, so required_approvals and the
|
||||
# required status check are both bypassable by exactly the account that does the merging.
|
||||
# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
|
||||
#
|
||||
# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
|
||||
# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
|
||||
# instead of probing names you assume exist. That mistake cost a wasted PATCH.
|
||||
#
|
||||
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
|
||||
# PR would never produce that context.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> updating branch protection on main"
|
||||
jq -n '{
|
||||
enable_push: false,
|
||||
enable_status_check: true,
|
||||
status_check_contexts: ["gate / gate (pull_request)"],
|
||||
required_approvals: 1,
|
||||
block_on_rejected_reviews: true,
|
||||
dismiss_stale_approvals: true,
|
||||
block_on_outdated_branch: false,
|
||||
block_admin_merge_override: true
|
||||
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
|
||||
|
||||
echo "==> verifying"
|
||||
api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
|
||||
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
|
||||
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
|
||||
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
|
||||
and rely on the status check alone.
|
||||
|
||||
Then prove it, rather than trusting the API:
|
||||
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
|
||||
git reset --hard HEAD~1
|
||||
NOTE
|
||||
Executable
+61
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env bash
|
||||
# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
|
||||
#
|
||||
# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
|
||||
# nowhere else. (It is briefly visible in this host's process list as a sops argument;
|
||||
# acceptable on a single-operator workstation, and far better than echoing it.)
|
||||
#
|
||||
# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
|
||||
echo "==> minting an instance-level registration token as '${U}'"
|
||||
|
||||
TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
echo "==> writing it into ${SECRET} via sops"
|
||||
DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
|
||||
sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
|
||||
unset TOKEN
|
||||
|
||||
echo "==> done. Verify the file is still encrypted (no plaintext token):"
|
||||
grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \
|
||||
|| { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
|
||||
|
||||
IMPORTANT -- a new token is usually NOT sufficient on its own.
|
||||
act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
|
||||
act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
|
||||
deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
|
||||
polls with an identity Gitea does not recognise -- so it never appears in the runner list,
|
||||
which is exactly the observed symptom (zero runners, pod running).
|
||||
|
||||
So after the Secret lands, force a genuine re-registration by clearing that state:
|
||||
|
||||
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
|
||||
kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
|
||||
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
|
||||
|
||||
(The PVC holds only runner registration state -- no build artifacts, nothing durable.
|
||||
Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
|
||||
|
||||
Then confirm it actually registered -- do not assume:
|
||||
|
||||
curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
|
||||
|
||||
A runner must appear with status "online" and the labels ubuntu-latest / homelab.
|
||||
NOTE
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
|
||||
# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
|
||||
#
|
||||
# The token value is NEVER printed. It goes from the token API into the secrets API and
|
||||
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
|
||||
#
|
||||
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
|
||||
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
|
||||
# confirm the check goes green rather than assuming the scope name.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
|
||||
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
|
||||
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
|
||||
SECRET=SETTINGS_API_TOKEN
|
||||
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> minting token '${NAME}' as ${U}"
|
||||
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
|
||||
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
|
||||
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
echo "==> storing it as repo Actions secret ${SECRET}"
|
||||
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
|
||||
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
|
||||
unset TOKEN
|
||||
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
|
||||
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
|
||||
rm -f /tmp/set-ci-token.out; exit 1
|
||||
fi
|
||||
rm -f /tmp/set-ci-token.out
|
||||
|
||||
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
|
||||
# reports success for a store that never happened -- which is exactly what it did.
|
||||
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
|
||||
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
|
||||
grep -q "$SECRET" <<<"$names" \
|
||||
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
|
||||
echo "==> stored and VERIFIED present: ${names}"
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Next:
|
||||
1. .gitea/workflows/gate.yml already passes this secret to the gate step.
|
||||
2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
|
||||
without the token, so a green run is the only proof the wiring works.
|
||||
3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
|
||||
instead of skipping. Do that only after you have seen it green, or you lock the repo.
|
||||
NOTE
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
|
||||
#
|
||||
# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
|
||||
# merges). This script does the mechanical half: prove the converted runner actually works,
|
||||
# and only then turn on enforcement.
|
||||
#
|
||||
# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
|
||||
# required status check while the runner is broken means NO PR can merge -- including the
|
||||
# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
|
||||
# only then does enforcement go on.
|
||||
#
|
||||
# Aborts at the first checkpoint that does not pass. Safe to re-run.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
|
||||
D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
|
||||
die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
|
||||
|
||||
say "PRECHECK — all four PRs must be merged first"
|
||||
for n in 8 5 6 7; do
|
||||
m=$(api "$A/pulls/$n" | jq -r .merged)
|
||||
printf ' PR #%s merged=%s\n' "$n" "$m"
|
||||
[ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
|
||||
done
|
||||
|
||||
say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
|
||||
|
||||
say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
|
||||
./infra/devcontainer/dev.sh run bash -c '
|
||||
kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
|
||||
echo "--- sidecar (initContainers) ---"
|
||||
kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
|
||||
-o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
|
||||
kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
|
||||
|| die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
|
||||
|
||||
say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
|
||||
n=0
|
||||
for i in $(seq 1 40); do
|
||||
n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
|
||||
printf '\r online runners: %s (%ss) ' "$n" "$((i*5))"
|
||||
[ "${n:-0}" -ge 1 ] && { echo; break; }
|
||||
sleep 5
|
||||
done
|
||||
[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
|
||||
|
||||
say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
|
||||
MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
|
||||
echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..."
|
||||
st=""
|
||||
for i in $(seq 1 60); do
|
||||
st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
|
||||
printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))"
|
||||
case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
|
||||
sleep 10
|
||||
done
|
||||
[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
|
||||
echo " the converted sidecar ran a real job green. Enforcement is now safe."
|
||||
|
||||
say "STEP 5 — require the gate + one approval"
|
||||
"$D/require-gate.sh"
|
||||
|
||||
say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
|
||||
git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
|
||||
git commit -q --allow-empty -m 'protection probe'
|
||||
if git push origin main 2>&1 | grep -q 'protected branch'; then
|
||||
echo " CONFIRMED: push to main rejected by the pre-receive hook."
|
||||
else
|
||||
echo " WARNING: the push was NOT rejected -- protection is not doing what it claims."
|
||||
fi
|
||||
git reset -q --hard origin/main
|
||||
|
||||
say "FINAL STATE"
|
||||
api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env bash
|
||||
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
|
||||
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
|
||||
# stopwatch so you can see the budget being spent.
|
||||
#
|
||||
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
|
||||
#
|
||||
# It never sees, prints, or handles the token: the file is already encrypted by sops.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
|
||||
T0=$(date +%s)
|
||||
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
# --- preflight: refuse to ship a file that is not what we think it is -------------
|
||||
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
|
||||
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
|
||||
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
|
||||
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
|
||||
# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
|
||||
# matching prefix fired a redundant run that serialised ahead of the PR run. Those
|
||||
# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
|
||||
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
|
||||
|
||||
el "committing on $BRANCH"
|
||||
git checkout -q -b "$BRANCH"
|
||||
git add "$SECRET"
|
||||
git commit -q -m 'chore(plex): fresh plex.tv claim token
|
||||
|
||||
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
|
||||
SHA=$(git rev-parse HEAD)
|
||||
|
||||
el "pushing"
|
||||
git push -q -u origin "$BRANCH"
|
||||
|
||||
el "opening PR"
|
||||
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
|
||||
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
|
||||
| api -X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
|
||||
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
|
||||
echo " PR #$PR"
|
||||
|
||||
el "waiting for the gate (expect ~1m40s)"
|
||||
for i in $(seq 1 60); do
|
||||
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
|
||||
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
|
||||
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
|
||||
sleep 5
|
||||
done
|
||||
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
|
||||
|
||||
el "merging PR #$PR"
|
||||
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
|
||||
git checkout -q main && git pull -q --ff-only origin main
|
||||
|
||||
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||
|
||||
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
|
||||
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
|
||||
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
|
||||
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
|
||||
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
|
||||
|
||||
el "verdict"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
|
||||
cat <<'NOTE'
|
||||
|
||||
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
|
||||
"Attempting..." with no success line -> token expired; retry, nothing is broken
|
||||
neither line -> already claimed, or env did not reach the container
|
||||
|
||||
Confirm it persisted (prints the attribute NAME only, never its value):
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
|
||||
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
|
||||
NOTE
|
||||
@@ -34,6 +34,14 @@ resources:
|
||||
# policies, the next import is blocked in the gap.
|
||||
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
|
||||
- ../platform/kyverno/admission/secret-provenance.yaml
|
||||
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
|
||||
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
|
||||
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
|
||||
# only ("kept as reference; THIS file is the buildable render root" in
|
||||
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
|
||||
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
|
||||
# leaving the conformance-trigger identity live with nothing bounding it.
|
||||
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
|
||||
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
|
||||
- cilium/l2-ip-pool.yaml
|
||||
- cilium/l2-announcement-policy.yaml
|
||||
|
||||
+317
-10
@@ -21,7 +21,7 @@ Cilium L2 LoadBalancer (game protocols authenticate players themselves).
|
||||
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
|
||||
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
|
||||
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
|
||||
| Game VIP | **192.168.1.55** (Cilium L2 LoadBalancer; `.53`=Gateway, `.54`=unifi, `.55`=crafty) |
|
||||
| Game VIP | `${CRAFTY_L4_VIP}` (Cilium L2 LoadBalancer) — **prod `192.168.2.22`** (live `EXTERNAL-IP`, measured 2026-09-29); base/dev `192.168.1.55` |
|
||||
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |
|
||||
|
||||
### fsGroup / user decision (the seerr lesson, applied)
|
||||
@@ -76,15 +76,16 @@ can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient
|
||||
## Game-port L4 exposure
|
||||
|
||||
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
|
||||
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP **192.168.1.55**
|
||||
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP `${CRAFTY_L4_VIP}` (**prod
|
||||
`192.168.2.22`**)
|
||||
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
|
||||
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
|
||||
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → 192.168.1.55`.
|
||||
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}`.
|
||||
|
||||
### Adding more game ports (when the operator creates servers)
|
||||
|
||||
The operator has **not created any servers yet**; the starter set covers 3 Java + 3 Bedrock slots. To
|
||||
add capacity:
|
||||
As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and
|
||||
Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:
|
||||
|
||||
1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
|
||||
2. Add that port to the matching Service in `service-l4.yaml` (one
|
||||
@@ -93,6 +94,274 @@ add capacity:
|
||||
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
|
||||
5. Commit → Flux reconciles.
|
||||
|
||||
### Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)
|
||||
|
||||
Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix.
|
||||
As of 2026-09-29 21:34 this server is **reachable on `19132/udp` and answering**, measured from
|
||||
off-cluster. Read both traps before changing anything.
|
||||
|
||||
#### Trap 1 — `transport=nethernet` (the shipped default) opens no RakNet UDP listener
|
||||
|
||||
Measured 20:54–21:32 against `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, DB
|
||||
`type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive):
|
||||
|
||||
- `/proc/net/udp` and `/proc/net/udp6` held **no socket on 19132** (`0x4ABC`) at all. Port 19132 was
|
||||
bound as **TCP** (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP.
|
||||
- That is nethernet working as designed, and the **shipped `bedrock_server_how_to.html`** — the
|
||||
vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet *"listens on a
|
||||
single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake, then
|
||||
negotiates a UDP connection with each client that carries the gameplay traffic."*
|
||||
- **So nethernet could never have worked here anyway, on two independent counts:** `service-l4.yaml`
|
||||
exposes 19132 **only in the UDP Service** (`crafty-l4-tcp` carries 25565-25567 and nothing else), and
|
||||
`crafty-allow-ingress` allows `19132 UDP` and **not** `19132 TCP`. The signaling port was unreachable
|
||||
at both the Service and the policy layer. The server's own `packet-statistics.txt` for that run
|
||||
agrees: `Total Received: 0, 0B` over 2293 s.
|
||||
- Crafty does not manage this property (`transport` appears nowhere under `/crafty/app/classes/`) and
|
||||
its *only* Bedrock probe is `raknet_ping.py`, so the panel logged `WARNING
|
||||
app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132`
|
||||
every ~5 s — **155 such lines** — against a server it had started successfully.
|
||||
|
||||
#### Trap 2 — under `transport=raknet` this build prints a scary ERROR banner and then works anyway
|
||||
|
||||
The operator set `transport=raknet` at 21:33:31 (`/crafty/logs/audit.log`: `Edited file
|
||||
/crafty/servers/c5837301-…/server.properties`) and started the server at 21:33:58. BDS **1.26.52**
|
||||
printed, at ERROR level:
|
||||
|
||||
```
|
||||
[ERROR] ================ TRANSPORT TYPE ERROR ===================
|
||||
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
|
||||
supported transport type.
|
||||
[ERROR] Players will not be able to connect to your game without NetherNet.
|
||||
[ERROR] To switch, set 'transport=nethernet' in server.properties.
|
||||
```
|
||||
|
||||
**That banner is advisory, not fatal. Do not revert `transport` on seeing it.** Every claim it makes
|
||||
about connectivity was false on this cluster within five seconds of it being printed:
|
||||
|
||||
- The process **kept running** (`./bedrock_server`, pid 47006) and **bound the RakNet sockets**:
|
||||
`/proc/net/udp` `00000000:4ABC` (19132, IPv4) and `/proc/net/udp6` `…:4ABD` (19133, IPv6), both owned
|
||||
by `bedrock_server` — `/proc/47006/fd/25` and `fd/26` → those socket inodes. The earlier **TCP** bind
|
||||
on `4ABC` is **gone**.
|
||||
- Crafty's warning **changed kind** at that instant: last `Got timeout` at `21:34:04,306`, first
|
||||
`Bedrock ping returned a string with too few fields` at `21:34:09,854`. That distinction is
|
||||
load-bearing — read `raknet_ping.py`: `Got timeout` is emitted *only* when `sock.recv` raises
|
||||
`TimeoutError` (nothing came back at all), whereas `too few fields` is reachable *only after* a
|
||||
`0x1C` Unconnected Pong has arrived **and its magic has validated**. The server is answering pings.
|
||||
- **End-to-end, from off-cluster:** a RakNet Unconnected Ping sent from the LAN to the prod VIP
|
||||
`192.168.2.22:19132` returns a 139-byte `0x1c` Unconnected Pong in ~12 ms, magic valid:
|
||||
|
||||
MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
|
||||
|
||||
Edition `MCPE`, protocol `2193`, version **`1.26.52`**, `0/10` players. **Players can reach this
|
||||
server now**, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.
|
||||
- The vendor doc on disk contradicts the banner and matches the observed behaviour. `transport` is
|
||||
documented as *"Allowed values: raknet, nethernet"*, and: *"**raknet** — The traditional Bedrock UDP
|
||||
transport. The server listens directly on `server-port` (IPv4) and `server-portv6` (IPv6), and
|
||||
clients connect to that address and port."* RakNet is also still fully compiled into the binary —
|
||||
`strings bedrock_server` yields `RakNetConnector`, `createUniqueRakPeer`, `RakPeerInterface` and the
|
||||
vendored build path `…/src-deps/raknet/raknet/RakNetSocket2.cpp`.
|
||||
|
||||
So *"NetherNet is the only supported transport type"* is a **support statement, not a capability
|
||||
statement**: Microsoft declines to support RakNet and ships it working. Treat the banner as a
|
||||
deprecation notice with an **unknown expiry** — a future BDS may genuinely remove RakNet, and that is
|
||||
the day this exposure needs GeyserMC (see *If RakNet is ever really removed* below).
|
||||
|
||||
> **The shipped-comment trap, stated plainly.** `server.properties` lists `raknet` as allowed and the
|
||||
> binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak
|
||||
> is neither prose: it is `/proc/net/udp` and a pong on the wire. Whenever this file says a Bedrock
|
||||
> server is or is not reachable, it is because someone pinged it — not because a comment said so.
|
||||
|
||||
#### The steady-state `too few fields` warning is benign and is now the expected signature
|
||||
|
||||
Crafty will log `Bedrock ping returned a string with too few fields - missing values [None]` every
|
||||
~5 s forever. **That means the ping SUCCEEDED.** BDS 1.26.52's pong carries 16 `;`-separated fields;
|
||||
Crafty 4.10.8's `pingpong_string_fields` table knows 14 and expects the last to be an empty
|
||||
terminator. It instead finds `'0'`, so `last_enumeration` stops at index 12 and the parser reports the
|
||||
one `None` terminator entry as missing. Every field that matters (edition, MOTD, protocol, version
|
||||
name, player count, max players, uuid, game mode, both ports) parsed correctly. **Do not chase this
|
||||
warning.** The failure signature is the *other* message, `Got timeout`.
|
||||
|
||||
### "You are not invited to play on this server" — the join gate is the allowlist, not the network (MEASURED 2026-09-29 21:56; **RESOLVED and confirmed 22:03**)
|
||||
|
||||
A Bedrock client reported **"Unable to connect / You are not invited to play on this server"**. That is
|
||||
an **authorization** refusal, and it is the expected, guaranteed behaviour of this server's current
|
||||
configuration. Measured:
|
||||
|
||||
| Property / file | Value | Source |
|
||||
|---|---|---|
|
||||
| `allow-list` | **`true`** (line 37) | `server.properties` |
|
||||
| `allowlist.json` | **`[]`** — 3 bytes, mtime 20:51, never edited since server create | `cat allowlist.json` |
|
||||
| `online-mode` | `true` (line 31) | `server.properties` |
|
||||
| `transport` | **`raknet`** (line 71), file mtime **still 21:33:31** | `server.properties` |
|
||||
| TCP 19132 listener | **none** (`NO-TCP-4ABC`) | `grep 4ABC /proc/net/tcp /proc/net/tcp6` |
|
||||
| UDP 19132 listener | present, `00000000:4ABC`, pid 47006 | `grep 4ABC /proc/net/udp` |
|
||||
|
||||
**`allow-list=true` with an empty `allowlist.json` refuses every player, unconditionally.** The shipped
|
||||
`bedrock_server_how_to.html` states it: *"If the `allow-list` property is enabled in server.properties
|
||||
then the server will only allow selected users to connect. To allow a user to connect you need to know
|
||||
their Xbox Live **Gamertag**."* An empty list selects nobody. This is not a network problem, not a
|
||||
transport problem, and not a platform problem — and no amount of Service or policy work will change it.
|
||||
|
||||
#### CONFIRMED: one gamertag added, and a player joined
|
||||
|
||||
**This is no longer a hypothesis. The fix was applied and it worked.** *(The join itself is
|
||||
**operator-reported** — 2026-09-29, the operator added the player's gamertag and the Bedrock client
|
||||
connected and joined. I did not watch it.)* Two independent things corroborate it **directly on disk**:
|
||||
|
||||
- `allowlist.json` went from `[]` to **one entry**, and that entry's **`xuid` is populated**
|
||||
(`keys=['ignoresPlayerLimit','name','xuid']`, file 75 bytes, mtime **22:03:17**). Per the vendor doc,
|
||||
`xuid` is *"Optional. If it's not set then it will be populated when someone with a matching name
|
||||
**connects**."* The operator wrote only the name — `audit.log` records `Edited file
|
||||
…/allowlist.json` at **22:02:13** — so **the XUID was written by the server, on connect, at 22:03:17**.
|
||||
That is server-authored evidence of an Xbox-Live-authenticated join. *(Observed: the file state and
|
||||
both timestamps. Reasoned from the vendor doc: that a populated XUID implies a connect.)*
|
||||
- The RakNet path still answers after the restart this involved: a pong from `192.168.2.22:19132` at
|
||||
22:06, `MCPE;…;1.26.52;0;10;…` (`players=0` — the player had left by then), new `server_guid` matching
|
||||
the new pid.
|
||||
|
||||
**So the causal chain is closed: `allow-list=true` + `allowlist.json == []` was the wall, and nothing
|
||||
else was.** The earlier mapping of the client's *"You are not invited"* string to the empty allowlist was
|
||||
**reasoned** from the vendor doc (there is no BDS log on disk to read it from); adding exactly one
|
||||
gamertag and getting exactly one successful join **confirms that mapping**. No transport change, no
|
||||
manifest change, and no network change was involved in the fix.
|
||||
|
||||
> **Note the route the operator actually used:** they **edited `allowlist.json` directly** and then
|
||||
> **restarted** the server (`audit.log`: edit 22:02:13 → `stop_server` 22:02:37 → `start_server`
|
||||
> 22:02:41; pid 47006 → **47875**). That is route 2 below, and a restart is a valid substitute for
|
||||
> `allowlist reload`. Route 1 (`allowlist add` in the console) avoids the restart and is still preferred.
|
||||
|
||||
**This is also positive evidence that the RakNet path works end to end.** A refusal authored by *our*
|
||||
server required the client to reach our server and get an answer back. The only listener on 19132 is
|
||||
the **RakNet UDP** socket — there is **no TCP 19132 listener at all**, and NetherNet requires a TCP
|
||||
signalling socket on `server-port`. So the client cannot have used NetherNet against this server; it
|
||||
arrived over `19132/udp`, through `crafty-l4-udp` and the VIP. **The 19132-19134/UDP exposure is not
|
||||
inert — it is the path that carried both the pong and this refusal.** *(Observed: the listeners, the
|
||||
absent TCP socket, `transport=raknet`. Reasoned from those: that the refusal traversed that path. Not
|
||||
captured: the actual player flow — see the caveat below.)*
|
||||
|
||||
#### Caveat: BDS console output is NOT on disk, so a join attempt cannot be correlated from the filesystem
|
||||
|
||||
`K3s-Bedrock`'s `log_path` in `crafty.sqlite` is the **empty string**, and `Dedicated_Server.txt` is
|
||||
**0 bytes**. Crafty does not persist a Bedrock server's stdout anywhere on the PVC — the console exists
|
||||
only in the panel UI. `commander.log` between 21:39 and 21:56 contains nothing but `homelab`
|
||||
crash-detection lines and the `raknet_ping` chatter: **no join, no gamertag, no XUID, no allowlist
|
||||
denial.** That absence is **not** evidence about where the refusal came from; it is evidence that
|
||||
Crafty logs no BDS stdout at all. **To see a rejected join you must watch the panel console live.**
|
||||
UNVERIFIED, and only observable in the UI.
|
||||
|
||||
#### How a Bedrock player actually gets authorized here
|
||||
|
||||
Identity is an **Xbox Live Gamertag** (`online-mode=true` forces XBL auth, which is what gives BDS a
|
||||
gamertag/XUID to check). `allowlist.json` is a JSON array; per the vendor doc the keys are `name` (the
|
||||
gamertag), optional `xuid` (*"populated when someone with a matching name connects"*), and
|
||||
`ignoresPlayerLimit`. So the operator adds **one entry per player, keyed on gamertag** — there is no
|
||||
group, role, or Keycloak integration, and **none of this touches the oauth2-proxy/Keycloak gate**,
|
||||
which protects only the web panel.
|
||||
|
||||
Three ways to do it, all **operator actions**. **Routes 1 and 2 are both proven** — route 2 is the one
|
||||
that produced the confirmed join above:
|
||||
|
||||
1. **Preferred — the panel console.** In Crafty, open `K3s-Bedrock` → Console and send:
|
||||
`allowlist add <Gamertag>` (quote it if it contains a space: `allowlist add "Example Name"`).
|
||||
This writes `allowlist.json` **and** takes effect immediately, with no restart.
|
||||
2. **Edit the file, then reload.** Write `/crafty/servers/c5837301-…/allowlist.json`, then send
|
||||
`allowlist reload` in the console — *"After you've modified the file you need to run the command
|
||||
`allowlist reload` to make sure that the server knows about your new change."* Shape:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "ignoresPlayerLimit": false, "name": "FirstPlayerGamertag" },
|
||||
{ "ignoresPlayerLimit": false, "name": "SecondPlayerGamertag" }
|
||||
]
|
||||
```
|
||||
|
||||
3. **Turn the allowlist off entirely** — `allow-list=false` in `server.properties` (restart), or
|
||||
`allowlist off` in the console. Note the vendor's warning: `allowlist on|off` *"does not change the
|
||||
value in the server.properties file"*, so a console `off` silently reverts on the next start.
|
||||
**Not recommended:** the game port is deliberately un-gated at the network layer (see *Game-port L4
|
||||
exposure*), so the allowlist is the **only** thing deciding who may join. Keep it `true` and enumerate
|
||||
players.
|
||||
|
||||
> **`allowlist.json` is application state on the PVC**, exactly like `transport=raknet`. Neither is in
|
||||
> git and **Flux can restore neither**. Be precise about what breaks them, because the two cases differ:
|
||||
>
|
||||
> - **A pod or server *restart* preserves both** — observed: the 22:02:48 restart (pid 47006 → 47875)
|
||||
> kept `transport=raknet` (`server.properties` mtime unchanged at 21:33:31) and kept the allowlist
|
||||
> entry. What a **pod** restart *does* break is that `K3s-Bedrock` has `auto_start=0`, so the Bedrock
|
||||
> server simply **does not come back** until someone starts it by hand.
|
||||
> - **A server *recreate* or reinstall silently resets both** — a fresh `server.properties` ships
|
||||
> `transport=nethernet` (nothing bound on 19132/udp) and a fresh `allowlist.json` is `[]` (everyone
|
||||
> refused). Neither change announces itself, and the server will look healthy and started in the panel
|
||||
> while being unreachable *and* unjoinable.
|
||||
>
|
||||
> **Treat both files as backup-relevant, and re-run the 5-step connect procedure above after any Bedrock
|
||||
> recreate or BDS update.**
|
||||
|
||||
#### Why this does not change the transport recommendation
|
||||
|
||||
`transport=raknet` is correct, and with the allowlist entry in place the combination is **proven
|
||||
end-to-end by a player who has joined** — not merely the best-reasoned option. The allowlist refusal was
|
||||
orthogonal to transport all along: it would have refused players identically under NetherNet. Switching
|
||||
to NetherNet would *add* work (TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus
|
||||
`server-udp-ports` pinned and exposed) and would admit **not one extra player**. The ranking below stands,
|
||||
and option 1 is now settled rather than provisional.
|
||||
|
||||
#### Decision (PROVEN, not provisional): `transport=raknet` stays; the 19132-19134/UDP exposure is correct and live
|
||||
|
||||
No manifest change is needed or wanted. `service-l4.yaml`, `networkpolicy.yaml` and the
|
||||
`containerPort`s were already shaped for exactly RakNet, and that shape is now **proven by a player who
|
||||
has connected and joined over `192.168.2.22:19132`** — not assumed, and not merely reasoned. The options
|
||||
that would have required manifest work are all **not recommended**:
|
||||
|
||||
| Option | Manifest consequence | Verdict |
|
||||
|---|---|---|
|
||||
| **Keep `transport=raknet`** + allowlist entries (**this is what works**) | **none** — the existing Service, policy and ports were already correct | **Done; proven.** A player has joined over the VIP. Zero manifest change, vanilla parity, real Bedrock world. |
|
||||
| **GeyserMC + Floodgate** on the `homelab` Paper server | none either (Geyser also binds `19132/udp`), but it would **collide** with the live BDS on 19132 — one of them must move | Hold in reserve. Only worth it if RakNet is really removed, or if you *want* one shared world. Cost: Bedrock players join the **Java** world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch. |
|
||||
| **Pin/downgrade BDS** to a pre-nethernet release | none | Unnecessary — the *current* build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all. |
|
||||
| **A third-party server** (PocketMine-MP, Nukkit) | none | No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring. |
|
||||
| **Accept NetherNet** | 19132-19134/UDP becomes **inert**; nethernet would instead need **TCP** 19132 added to `crafty-l4-tcp` *and* to `crafty-allow-ingress`, plus `server-udp-ports` pinned and exposed for the negotiated media path | No — and note it is **not currently in use**: `transport=raknet` and there is no TCP 19132 listener. Strictly more manifest work than the option that already works, it admits no extra player (the allowlist is the gate), and WebRTC media behind a shared VIP and SNAT is unproven here. |
|
||||
|
||||
#### If RakNet is ever really removed
|
||||
|
||||
Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not
|
||||
a config one. The migration is **GeyserMC + Floodgate as a plugin on the existing `homelab` Paper
|
||||
server**, because Geyser speaks RakNet on `19132/udp` itself: the VIP, the Service, the policy and the
|
||||
`containerPort`s all stay **byte-for-byte as they are**, and Bedrock clients keep connecting to
|
||||
`crafty-mc.home.keithjackson.dev:19132`. What changes is the world: Bedrock players land in the Java
|
||||
world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server
|
||||
is retired, and `K3s-Bedrock` stops the day Geyser starts — **they cannot share 19132**. Installing it
|
||||
is a PVC write plus a Paper restart, i.e. an **operator action**: drop `Geyser-Spigot.jar` and
|
||||
`floodgate-spigot.jar` into `/crafty/servers/13c7b85d-…/plugins/` and restart `homelab`, then set
|
||||
Geyser's `bedrock.port: 19132` and `clone-remote-port: false` in its generated `config.yml`.
|
||||
Geyser's own Bedrock-version support is **external and unmeasured here** — check that it supports the
|
||||
Bedrock client version in use before committing.
|
||||
|
||||
### Bedrock realities that are NOT platform defects but present identically
|
||||
|
||||
- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132
|
||||
inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it
|
||||
will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the
|
||||
address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**.
|
||||
`external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and
|
||||
is correct for Bedrock too — only the port differs from Java's 25565.
|
||||
- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no
|
||||
arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS
|
||||
redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`.
|
||||
- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2 — now
|
||||
OBSERVED, not merely documented.** Under `transport=raknet` it makes a server bind the **default**
|
||||
ports 19132 *and* 19133 even when `server-port`/`server-portv6` say otherwise; the shipped
|
||||
`bedrock_server_how_to.html` says so, and on 2026-09-29 a RakNet ping to
|
||||
`192.168.2.22:19133` returned **the same pong, from the same `server_guid`, as `:19132`** — one
|
||||
server occupying two of the three Bedrock slots. `:19134` correctly timed out. So a second Bedrock
|
||||
server put on 19133 will fight the first. Set `enable-lan-visibility=false` on every Bedrock server
|
||||
after the first; LAN discovery cannot work here anyway (above).
|
||||
- **`server-portv6` is the IPv6 listener and is not exposed; `19133` is nonetheless in use.**
|
||||
`service-l4.yaml` is IPv4 single-stack, so the *IPv6* socket `…:4ABD` that BDS opens for
|
||||
`server-portv6=19133` is unreachable from the LAN. But because `enable-lan-visibility=true` (above)
|
||||
the server *also* answers on **IPv4 19133**, which the UDP Service does expose — measured. So
|
||||
`19133` in `server.properties` genuinely does mean "slot 1 is taken", contrary to what this file
|
||||
previously said.
|
||||
|
||||
## Policies
|
||||
|
||||
`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
|
||||
@@ -114,14 +383,48 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
|
||||
local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
|
||||
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
|
||||
`/crafty/app/config` and `/crafty/servers` without EACCES.
|
||||
3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
|
||||
connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
|
||||
from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
|
||||
3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`;
|
||||
start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or
|
||||
`192.168.2.22:25565`) from the LAN and confirm the world loads.
|
||||
4. **Bedrock: the connect procedure (VERIFIED end-to-end 2026-09-29 — a player has joined).**
|
||||
Creating and starting the server is **not sufficient**. Do these in order; each step has a distinct
|
||||
failure signature:
|
||||
|
||||
1. **`transport=raknet`** in `server.properties`. The vendor default is `nethernet`, which opens
|
||||
**no UDP listener at all**.
|
||||
2. **A live UDP listener:** `grep 4ABC /proc/net/udp /proc/net/udp6` → a wildcard bind. Under
|
||||
`nethernet` you instead see `4ABC` on **TCP**, which nothing here exposes.
|
||||
3. **A pong from off-cluster:** a RakNet Unconnected Ping to `192.168.2.22:19132` returns a `0x1c`
|
||||
packet whose MOTD string starts `MCPE;`. This proves the VIP, Service and policy path.
|
||||
4. **An allowlist entry for the player's Xbox Gamertag.** `allow-list=true` with
|
||||
`allowlist.json == []` refuses **every** player with *"You are not invited to play on this
|
||||
server"* while the server is otherwise perfectly reachable. In the panel console:
|
||||
`allowlist add <Gamertag>`.
|
||||
5. **Add Server** by hand in a Windows/mobile client: `crafty-mc.home.keithjackson.dev`, port
|
||||
`19132`. It will never appear in the LAN/Friends tab, and a console cannot reach it at all.
|
||||
|
||||
**Two log messages appear on a fully working server and must be ignored:** BDS's
|
||||
`================ TRANSPORT TYPE ERROR ===================` banner (advisory — see *Trap 2*), and
|
||||
Crafty's perpetual `Bedrock ping returned a string with too few fields`. Neither indicates a fault;
|
||||
chasing them is what cost two sessions. The **real** failure signatures are `Got timeout while
|
||||
issuing bedrock ping` (no listener) and *"You are not invited"* (allowlist).
|
||||
|
||||
## Runtime state: the servers themselves are NOT in git
|
||||
|
||||
As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
|
||||
on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
|
||||
As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`):
|
||||
|
||||
| Name | Type | Port | UUID | `auto_start` |
|
||||
|---|---|---|---|---|
|
||||
| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** |
|
||||
| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ |
|
||||
|
||||
`K3s-Bedrock` was created 2026-09-29 20:51 and **still** has `auto_start=0` **and**
|
||||
`crash_detection=0` — re-measured 2026-09-29 21:41 straight out of `crafty.sqlite` (read-only), so the
|
||||
create-API trap documented at the bottom of this file is still live and still unfixed. **It will not
|
||||
come back after a pod restart**, and merging any change to `deployment.yaml` restarts the pod
|
||||
(`strategy: Recreate`). PATCH both to true and read them back.
|
||||
|
||||
`homelab` was created through Crafty's v2 API, and **Mojang's
|
||||
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
|
||||
DECISIONS-LOG entry of that date.
|
||||
|
||||
@@ -132,6 +435,10 @@ application state on the `crafty-data` Longhorn PVC, in two places, and **Flux c
|
||||
heap sizes, `auto_start` / `crash_detection`.
|
||||
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
|
||||
**the generated world**. The jar is re-downloadable; the world is not.
|
||||
**`transport=raknet` lives here too.** It is the one setting that makes Bedrock reachable at all (see
|
||||
*the transport trap*), it is not in git, Flux cannot restore it, and a freshly created or reinstalled
|
||||
Bedrock server gets the vendor default `transport=nethernet` back — i.e. it silently regresses to
|
||||
"nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.
|
||||
|
||||
So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
|
||||
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
|
||||
|
||||
@@ -26,10 +26,94 @@ spec:
|
||||
provider:
|
||||
name: cloudflare
|
||||
# Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
|
||||
#
|
||||
# BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block
|
||||
# below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list.
|
||||
sources:
|
||||
- gateway-httproute
|
||||
- service
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14)
|
||||
#
|
||||
# THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's
|
||||
# address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
||||
# public internet and that Pi-hole serves the internal split-horizon answers; the renderer
|
||||
# contract said an `internal` service renders "no public DNS record". Neither sentence was
|
||||
# enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent,
|
||||
# radarr, semaphore, sonarr) published `<name>.home.keithjackson.dev` into this PUBLIC
|
||||
# authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP
|
||||
# and a complete service inventory to anyone who could guess a label, and made every one of
|
||||
# those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1
|
||||
# strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own
|
||||
# services. Pre-state captured at the AUTHORITY and committed:
|
||||
# .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
||||
#
|
||||
# THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`.
|
||||
# An `internal` service's rendered HTTPRoute carries
|
||||
# external-dns.alpha.kubernetes.io/controller: none
|
||||
# ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image
|
||||
# registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller`
|
||||
# annotation is present and != its own `dns-controller`:
|
||||
# source/annotations/processors.go IsControllerMismatch()
|
||||
# source/gateway.go Endpoints(): if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
||||
# source/service.go Endpoints(): the identical check
|
||||
# The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an
|
||||
# endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway
|
||||
# hostname and no existing record at risk: an annotated route published nothing at all, and
|
||||
# ADDING the annotation to a route that already had a record produced
|
||||
# action=DELETE record=<name> type=A and action=DELETE record=a-<name> type=TXT
|
||||
# one ~60s reconcile later, and moved nothing else in the zone.
|
||||
#
|
||||
# WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter`
|
||||
# (--label-filter) is the stronger default posture — publish only what is labelled — and
|
||||
# v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by
|
||||
# source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here
|
||||
# because its blast radius is not this defect's. Nine publishing objects in this repo are
|
||||
# hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and
|
||||
# seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp,
|
||||
# the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile,
|
||||
# and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a
|
||||
# renderer one. They are enumerated in the shrink-only register in
|
||||
# conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set
|
||||
# cannot grow silently — adjudicate them there, then this filter becomes the right hardening.
|
||||
#
|
||||
# DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
|
||||
#
|
||||
# The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
|
||||
# that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
|
||||
# actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
|
||||
# this change, for two independent reasons:
|
||||
#
|
||||
# 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
|
||||
# (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
|
||||
# points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
|
||||
# certificate. Whether prod's public services belong behind it, behind a new forward, or
|
||||
# are not meant to be public at all is an operator decision. Breaking five services' DNS
|
||||
# on a guess is worse than leaving a known-wrong-but-stable state.
|
||||
# 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
|
||||
# real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
|
||||
# GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
|
||||
# no per-service target override while every public service shares the `shared` Gateway.
|
||||
# Annotating the Gateway would move EVERY gateway-sourced record, which after this change
|
||||
# still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
|
||||
# routable address for admin UIs. That is strictly worse than the status quo.
|
||||
#
|
||||
# SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
|
||||
# is answered, because a knob that cannot take effect invites someone to set it and expect one:
|
||||
# (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
|
||||
# port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
|
||||
# (b) HOW should they be separated from the non-public names that share the Gateway? Options,
|
||||
# roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
|
||||
# annotation, with only the public services' routes attached; or publishing the five names
|
||||
# from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
|
||||
# which restores per-service targeting without a second Gateway.
|
||||
# Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
# On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
# Serves: R-SEC-02 (CI holds no broad cluster access)
|
||||
#
|
||||
# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
|
||||
#
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
|
||||
# `create jobs` in namespace `conformance` can therefore create a Job that runs as
|
||||
# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
|
||||
# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
|
||||
# privileged identity unless something constrains the SHAPE of what it may create.
|
||||
#
|
||||
# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
|
||||
# committed conformance run and deny everything else. The operator chose this model
|
||||
# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
|
||||
# GitOps-native alternative.
|
||||
#
|
||||
# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
|
||||
# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
|
||||
# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
|
||||
# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
|
||||
# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
|
||||
#
|
||||
# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
|
||||
# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
|
||||
#
|
||||
# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
|
||||
# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
|
||||
# enforceable half: the image must live under the conformance repository path AND be
|
||||
# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
|
||||
# registry path could publish a different digest. TIGHTEN THIS when the image is built by
|
||||
# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
|
||||
# PR, which is exactly where a change to the security boundary belongs.
|
||||
apiVersion: kyverno.io/v1
|
||||
kind: ClusterPolicy
|
||||
metadata:
|
||||
name: conformance-trigger-job-shape
|
||||
annotations:
|
||||
policies.kyverno.io/title: Constrain what the conformance CI trigger may create
|
||||
policies.kyverno.io/category: Supply chain
|
||||
policies.kyverno.io/severity: critical
|
||||
homelab.keithjackson.dev/requirements: R-SEC-02
|
||||
homelab.keithjackson.dev/scope: admission-only
|
||||
spec:
|
||||
validationFailureAction: Enforce
|
||||
failurePolicy: Fail
|
||||
background: false
|
||||
rules:
|
||||
- name: trigger-may-create-only-the-committed-conformance-run
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger may create ONLY the committed conformance run: a single container,
|
||||
digest-pinned conformance-runner image, serviceAccountName conformance-runner,
|
||||
command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
|
||||
and only the `workspace` and `results` volumes. See
|
||||
platform/kyverno/admission/conformance-trigger-job-shape.yaml.
|
||||
pattern:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
=(hostNetwork): "false"
|
||||
=(hostPID): "false"
|
||||
=(hostIPC): "false"
|
||||
containers:
|
||||
- image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
|
||||
command:
|
||||
- "./run.sh"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
=(privileged): "false"
|
||||
|
||||
- name: trigger-volumes-must-be-workspace-or-results
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
|
||||
volume is how a constrained pod reaches a host path, a secret, or another SA's token.
|
||||
# foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
|
||||
# matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
|
||||
# asserts EVERY volume is one of the two allowed names.
|
||||
foreach:
|
||||
- list: "request.object.spec.template.spec.volumes"
|
||||
deny:
|
||||
conditions:
|
||||
all:
|
||||
- key: "{{ element.name }}"
|
||||
operator: AnyNotIn
|
||||
value: ["workspace", "results"]
|
||||
|
||||
- name: trigger-args-must-be-the-committed-flags
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger Job args may contain only the conformance run's own flags
|
||||
(--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
|
||||
Anything else — a shell, a path, a redirect — is denied.
|
||||
foreach:
|
||||
- list: "request.object.spec.template.spec.containers[0].args"
|
||||
deny:
|
||||
conditions:
|
||||
all:
|
||||
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
|
||||
operator: Equals
|
||||
value: false
|
||||
@@ -20,3 +20,7 @@ resources:
|
||||
# --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
|
||||
# (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
|
||||
- secret-provenance.yaml
|
||||
# --- R-SEC-02: constrain what the conformance CI trigger may create ---
|
||||
# THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
|
||||
# subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
|
||||
- conformance-trigger-job-shape.yaml
|
||||
@@ -0,0 +1,57 @@
|
||||
# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
|
||||
#
|
||||
# Each DENY case below is a real escalation path, not a style violation. The one that matters
|
||||
# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
|
||||
# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
|
||||
# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
|
||||
# this policy.
|
||||
#
|
||||
# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
|
||||
# never fires and every case trivially passes — a test that proves nothing.
|
||||
apiVersion: cli.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: conformance-trigger-job-shape
|
||||
policies:
|
||||
- ../../admission/conformance-trigger-job-shape.yaml
|
||||
resources:
|
||||
- resources.yaml
|
||||
userinfo: user-info.yaml
|
||||
results:
|
||||
# ---- the committed run is ADMITTED ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-may-create-only-the-committed-conformance-run
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-volumes-must-be-workspace-or-results
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-args-must-be-the-committed-flags
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
|
||||
# ---- THE escalation: runner SA + arbitrary command ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-may-create-only-the-committed-conformance-run
|
||||
kind: Job
|
||||
resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
|
||||
result: fail
|
||||
|
||||
# ---- an extra volume is the way out of a constrained pod ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-volumes-must-be-workspace-or-results
|
||||
kind: Job
|
||||
resources: [extra-volume]
|
||||
result: fail
|
||||
|
||||
# ---- args beyond the committed flags ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-args-must-be-the-committed-flags
|
||||
kind: Job
|
||||
resources: [injected-args]
|
||||
result: fail
|
||||
@@ -0,0 +1,154 @@
|
||||
# The committed shape — must be ADMITTED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: good-run
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
- name: results
|
||||
emptyDir: {}
|
||||
---
|
||||
# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evil-command
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# An arbitrary image running as the privileged SA. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evil-image
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: docker.io/library/alpine:latest
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: tag-not-digest
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# Host namespaces + privilege escalation. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: host-namespaces
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
hostPID: true
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: false
|
||||
allowPrivilegeEscalation: true
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# An extra volume — the way out of a constrained pod. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: extra-volume
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
- name: hostroot
|
||||
hostPath:
|
||||
path: /
|
||||
---
|
||||
# Injected args beyond the committed flags. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: injected-args
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
args: ["--layer", "l2", ";", "curl", "http://evil/"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
@@ -0,0 +1,12 @@
|
||||
# The requesting identity the policy keys on. Without this the `subjects` match never fires and
|
||||
# every case below would trivially "pass" — proving nothing.
|
||||
apiVersion: cli.kyverno.io/v1alpha1
|
||||
kind: UserInfo
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
userInfo:
|
||||
username: system:serviceaccount:conformance:conformance-trigger
|
||||
groups:
|
||||
- system:serviceaccounts
|
||||
- system:serviceaccounts:conformance
|
||||
- system:authenticated
|
||||
@@ -904,6 +904,132 @@ spec:
|
||||
summary: "Egress VPN tunnel down"
|
||||
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
|
||||
#
|
||||
# `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
|
||||
# having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
|
||||
# exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
|
||||
# re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
|
||||
# at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
|
||||
# 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
|
||||
# carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
|
||||
# exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
|
||||
#
|
||||
# The datapath now self-heals: `roles/vpn_egress_gateway` publishes
|
||||
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
|
||||
# OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
|
||||
# and that node write is what makes Cilium re-resolve. These two rules watch whether it
|
||||
# ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
|
||||
#
|
||||
# WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
|
||||
# `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
|
||||
# observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
|
||||
# * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
|
||||
# `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
|
||||
# there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
|
||||
# metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
|
||||
# `cilium_*` and `hubble_*` query returns an empty vector today (which also means
|
||||
# `HubbleDeniedFlowSpike` above cannot fire yet).
|
||||
# * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
|
||||
# CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
|
||||
# map, not an exporter. No metric name is invented here to paper over that.
|
||||
# So the observable is the label: the input to the re-steer, compared against the live tun0
|
||||
# identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
|
||||
# ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
|
||||
# `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
|
||||
#
|
||||
# WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
|
||||
# firing, the label write failing or never being deployed, the convergence timer being dead —
|
||||
# i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
|
||||
# "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
|
||||
# until either the re-steer script exports the map's ifindex through a node-exporter textfile
|
||||
# collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
|
||||
# A probe was considered and rejected for now: there is no blackbox-exporter and no probe
|
||||
# CronJob in this repo, so it means a NEW always-on component whose own failures become a new
|
||||
# alerting surface — worth doing, but not as the thing that closes a hole today.
|
||||
#
|
||||
# WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
|
||||
# VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
|
||||
# label-to-number conversion, so the level-triggered equality test is not expressible. What IS
|
||||
# expressible is the same invariant counted over a window: the published label must change at
|
||||
# least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
|
||||
# is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
|
||||
# that held still, 2 for one that changed value inside the window); the `max by (<label>)`
|
||||
# wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
|
||||
# restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
|
||||
# which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
|
||||
#
|
||||
# Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
|
||||
# +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
|
||||
# trade against the repo's alert-fatigue history (an unbounded critical on the one channel
|
||||
# that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
|
||||
# email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
|
||||
- alert: VPNEgressSteeringStale
|
||||
# FIRES when the live tun0 was re-created more times in the last hour than the gateway
|
||||
# node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
|
||||
# tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
|
||||
# and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
|
||||
# seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
|
||||
# cold boot), so `for: 15m` cannot fire on a healthy reconnect.
|
||||
expr: |
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(
|
||||
count(
|
||||
count_over_time(
|
||||
max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
|
||||
)[1h:1m]
|
||||
)
|
||||
) - 1
|
||||
)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare
|
||||
`kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
|
||||
`cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
|
||||
(or setting any node label) forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
- alert: VPNEgressUplinkLabelMissing
|
||||
# The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
|
||||
# in kube-state-metrics, and if that label is missing its comparison degrades to an empty
|
||||
# vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
|
||||
# node with no published uplink means the re-steer mechanism is not in place at all
|
||||
# (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
|
||||
# kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
|
||||
# the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
|
||||
# rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
|
||||
# that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
|
||||
# on the way in, and so an Ansible run in progress does not either.
|
||||
expr: |
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
and on()
|
||||
absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
|
||||
nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
|
||||
`VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -231,7 +231,7 @@ spec:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
@@ -116,10 +116,14 @@ fi
|
||||
# `.git` file points outside the mount, so git cannot read the index there — and with
|
||||
# `pipefail` that failure silently emptied the discovery, i.e. a green run over ZERO tests
|
||||
# (observed 2026-09-12). `find` needs no repo metadata, so discovery cannot be disabled by
|
||||
# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up.
|
||||
# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up,
|
||||
# and .claude/ with it: .claude/worktrees/agent-*/ holds full repo copies for isolated agent
|
||||
# runs, so an unpruned find reported the SAME manifest under test once per worktree (up to 18
|
||||
# times), and the L0 check that compares this listing against the tree failed on paths that
|
||||
# are not authored surface at all (.claude/ is gitignored, .gitignore:43).
|
||||
discover() {
|
||||
# Companion convention, anywhere in the tree.
|
||||
find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
|
||||
find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
|
||||
# Suite convention, ANYWHERE in the tree — not just this directory.
|
||||
#
|
||||
# This used to be `find "${HERE}" -maxdepth 1`, which made the suite convention a
|
||||
@@ -129,7 +133,7 @@ discover() {
|
||||
# as the rules this harness exists to test. Found 2026-09-13 adding apps/wave-*/tests/. Scoped by
|
||||
# NAME only; manifests_for() then refuses (loudly) any *.test.yaml that is not inside a tests/
|
||||
# directory, so a misplaced file fails rather than disappearing.
|
||||
find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.test.yaml' -print 2>/dev/null
|
||||
find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.test.yaml' -print 2>/dev/null
|
||||
}
|
||||
|
||||
TESTS=()
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
# promtool rule-unit tests for VPNEgressSteeringStale + VPNEgressUplinkLabelMissing
|
||||
# (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
|
||||
# `egress-class=vpn` pods leave through one gateway node's tun0. Cilium resolves that interface to an
|
||||
# IFINDEX and caches it; an OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the
|
||||
# gateway node's up-script publishes `homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex>` so that
|
||||
# node write forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a
|
||||
# destroyed interface and every vpn-class pod is dropped — while tun0 itself stays up, so VPNTunnelDown
|
||||
# says nothing. That shape ran ~80 minutes undetected on the dev gateway.
|
||||
#
|
||||
# VPNEgressSteeringStale detects it by comparing HOW MANY TIMES tun0's ifindex changed in the last hour
|
||||
# against HOW MANY DISTINCT `egress-uplink` label values appeared in the same hour. The subtlety worth
|
||||
# testing is that a label VALUE change is not a sample change: it RETIRES one series and STARTS another,
|
||||
# so "the label followed the tunnel" shows up as two overlapping series in the window, not as a
|
||||
# changing value. Tests 1 and 2 are the same tunnel reconnect with and without that second series, so
|
||||
# they isolate exactly that mechanism — anything that broke the series-counting half would make test 2
|
||||
# fire, and anything that broke the tun0 half would make test 1 quiet.
|
||||
#
|
||||
# VPNEgressUplinkLabelMissing is the WATCHDOG for the rule above: with no uplink label anywhere, the
|
||||
# staleness comparison has nothing to compare and is structurally INERT — present, evaluating, and
|
||||
# incapable of firing. Tests 3 and 4 pin that, because an inert critical alert is the failure mode this
|
||||
# whole group exists to prevent.
|
||||
#
|
||||
# Fixture values are prod's real shape: gateway k3s-prod-3, tun0 ifindex 139 with address
|
||||
# 10.150.192.86 (live label value `10.150.192.86-139`), reconnecting to ifindex 140.
|
||||
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. THE BUG — tun0 was re-created, the label did NOT follow. Steering is stale: FIRES.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect with no label update - VPNEgressSteeringStale fires
|
||||
input_series:
|
||||
# tun0 re-created at t=41m: ifindex 139 -> 140. One change inside the trailing hour.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
# tun0 is UP and passing traffic the whole time — this is what makes the failure deceptive, and
|
||||
# it is asserted below: VPNTunnelDown stays QUIET while egress is actually broken.
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# The node label NEVER changes, so only ONE distinct uplink value exists in the window.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x80'
|
||||
alert_rule_test:
|
||||
# 1 tunnel change > (1 distinct label value - 1) = 0, held for 15m.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare `kubectl get nodes -L
|
||||
homelab.keithjackson.dev/egress-uplink` against `cilium-dbg bpf egress list` on the
|
||||
gateway node; re-applying the CiliumEgressGatewayPolicy (or setting any node label)
|
||||
forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Not yet: at 45m the condition has been true only ~4m, under the 15m `for`.
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
# The watchdog must stay quiet — the label IS present here.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# THE HEADLINE CLAIM, asserted rather than asserted-in-prose: the tunnel alert says NOTHING
|
||||
# while vpn-class egress is completely broken. If VPNTunnelDown covered this failure, the
|
||||
# staleness rule would not need to exist.
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE HEALTHY RECONNECT — same tunnel change, but the label followed it. STAYS QUIET.
|
||||
# This is the half that proves the rule is not simply "any reconnect alerts".
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect WITH the label following - stays quiet
|
||||
input_series:
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# A label VALUE change retires one series and starts another. Both are inside the trailing
|
||||
# hour, so the window holds TWO distinct uplink values: 2 - 1 = 1, which is NOT < 1 change.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x40 _x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-140"}'
|
||||
values: '_x41 1+0x39'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 75m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 3. THE INERT-ALERT CASE — a gateway node publishing NO uplink label at all.
|
||||
# VPNEgressSteeringStale cannot fire in this state, which is exactly why this warning exists.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: gateway node with no egress-uplink label - VPNEgressUplinkLabelMissing fires
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true"}'
|
||||
values: '1+0x60'
|
||||
# tun0 is reconnecting the whole time. The staleness rule STILL cannot fire, because it has no
|
||||
# label series to compare against — that is the inertness being alerted on.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x30 140+0x30'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 35m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without
|
||||
it nothing can tell whether Cilium's egress steering followed the last tunnel reconnect,
|
||||
and `VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Under the 30m `for`.
|
||||
- eval_time: 20m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# And the proof of inertness: the critical rule is silent despite a real reconnect.
|
||||
- eval_time: 55m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 55m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. NO GATEWAY NODE AT ALL — neither rule may fire. Both are guarded by an `and on()` on the
|
||||
# egress-gateway label so they stay silent on a cluster with no VPN gateway, rather than
|
||||
# alerting forever about an absent feature.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: no egress-gateway node - both rules silent
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.10:9100",device="eth0"}'
|
||||
values: '2+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
@@ -24,6 +24,36 @@
|
||||
# pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
|
||||
# its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
|
||||
#
|
||||
# RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds
|
||||
# no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up
|
||||
# cannot protect anything, and ATTEMPTING it actively breaks the backup.
|
||||
#
|
||||
# HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced
|
||||
# mid-flight the PVB fails with `error to expose PVB: error getting client pod <pod>: pods "<pod>"
|
||||
# not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not
|
||||
# advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a
|
||||
# backup that protected every real volume. Any Deployment rollout landing in the backup window is
|
||||
# enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is
|
||||
# how a real backup failure gets ignored, and this one guards R-STATE-03.
|
||||
#
|
||||
# OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a
|
||||
# targeted exclusion:
|
||||
# * ns velero — Velero's OWN transient kopia repository-maintenance Jobs
|
||||
# (`<ns>-default-kopia-maintain-job-<ts>`), emptyDir `plugins` + `scratch`. Four
|
||||
# failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed,
|
||||
# 2 errors, 3079 of 3079 items backed up.
|
||||
# * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs
|
||||
# writable scratch). Four failed PVBs when a runbook edit rolled the Deployment
|
||||
# while the backup was running.
|
||||
#
|
||||
# PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open
|
||||
# question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a
|
||||
# THROWAWAY ConfigMap first, leaving this one untouched until it passed:
|
||||
# control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed
|
||||
# same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
|
||||
# The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted.
|
||||
#
|
||||
|
||||
# `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
|
||||
# narrow it; deliberately not, so a second export is covered too). First matching rule wins.
|
||||
#
|
||||
@@ -44,6 +74,12 @@ data:
|
||||
policies.yaml: |
|
||||
version: v1
|
||||
volumePolicies:
|
||||
# emptyDir first: cheapest condition, and the most common match.
|
||||
- conditions:
|
||||
volumeTypes:
|
||||
- emptyDir
|
||||
action:
|
||||
type: skip
|
||||
- conditions:
|
||||
nfs: {}
|
||||
action:
|
||||
|
||||
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
||||
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
||||
gateway node is the next stop.
|
||||
|
||||
### And now something watches it (2026-09-12)
|
||||
|
||||
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
|
||||
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
|
||||
|
||||
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
|
||||
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
|
||||
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
|
||||
steering pointed at a destroyed ifindex.
|
||||
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
|
||||
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
|
||||
in which the rule above would be inert. The watchdog for the watchdog.
|
||||
|
||||
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
|
||||
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
|
||||
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
|
||||
the label to exist as a metric at all — the two changes must ship together.
|
||||
|
||||
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
|
||||
exposes no metric carrying the egress map's contents (the map is readable only through
|
||||
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
|
||||
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
|
||||
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
|
||||
probe from a vpn-class pod. Neither exists yet.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
||||
|
||||
+14
-4
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
|
||||
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
||||
|
||||
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
|
||||
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
|
||||
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
||||
help. These pages are what makes the link resolve.
|
||||
|
||||
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
||||
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
||||
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
|
||||
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
|
||||
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
|
||||
|
||||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||||
|
||||
@@ -84,8 +85,10 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
|
||||
|
||||
### Networking
|
||||
- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
|
||||
cannot see, and what the kill-switch actually does.
|
||||
- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
|
||||
egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
|
||||
- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
|
||||
ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
|
||||
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
||||
|
||||
### Backups and DR
|
||||
@@ -106,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||||
every green signal missed.
|
||||
|
||||
### Operator procedures (no alert links here)
|
||||
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
|
||||
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
|
||||
### Virtualisation
|
||||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||||
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
Loaded 100 of 130 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user