Compare commits
66
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8c7a770de2 | ||
|
|
1c5974013e | ||
|
|
4791d4a43a | ||
|
|
dfd608ac32 | ||
|
|
1d2d9d6549 | ||
|
|
4b31f97486 | ||
|
|
50a48e6f07 | ||
|
|
911ead5fbd | ||
|
|
8eb5baffa4 | ||
|
|
4634b44cc7 | ||
|
|
6586919132 | ||
|
|
e6f6e4febb | ||
|
|
e8e6b9c681 | ||
|
|
fcc66c7858 | ||
|
|
b9e8a4f9e8 | ||
|
|
b2d9883958 | ||
|
|
00efd4b828 | ||
|
|
67a54a33a4 | ||
|
|
db1a623073 | ||
|
|
ce271225de | ||
|
|
e72050c690 | ||
|
|
761e25cc3f | ||
|
|
e7508931ca | ||
|
|
bf98a99af6 | ||
|
|
8e017fad91 | ||
|
|
27d0ef3324 | ||
|
|
bbd4f682a6 | ||
|
|
0391139240 | ||
|
|
e9f441b5ce | ||
|
|
234cc6a71a | ||
|
|
69a0d1a1c7 | ||
|
|
d3a0d587a4 | ||
|
|
fe862ab527 | ||
|
|
a2665c0511 | ||
|
|
5a9e455149 | ||
|
|
4fd75ccdaa | ||
|
|
d7f24d0a9e | ||
|
|
712f4c9336 | ||
|
|
e3e56af461 | ||
|
|
a3f4ddb971 | ||
|
|
5f3ea63b7f | ||
|
|
23c335257b | ||
|
|
9339d5a79e | ||
|
|
a1cc0a5556 | ||
|
|
6eaaaae334 | ||
|
|
cde02d19d2 | ||
|
|
05a8094785 | ||
|
|
d99d47494c | ||
|
|
6f231a4311 | ||
|
|
623fa14256 | ||
|
|
16cbd4a79e | ||
|
|
755c08ab40 | ||
|
|
b389566641 | ||
|
|
7887ecbbdc | ||
|
|
d717d8e979 | ||
|
|
bc106caac1 | ||
|
|
79c703bc79 | ||
|
|
3523617671 | ||
|
|
6a4dc0ada7 | ||
|
|
14d5ccbadd | ||
|
|
05881d5a6a | ||
|
|
cf0d6c6beb | ||
|
|
20a28d0993 | ||
|
|
b184a42729 | ||
|
|
4080065aa7 | ||
|
|
16c6234696 |
No files matched your search
+28
-14
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
|
||||
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
|
||||
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
|
||||
|
||||
## Honest SKIP — never fake green (Phase 0)
|
||||
## Honest SKIP — never fake green
|
||||
|
||||
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
|
||||
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
|
||||
skip logic below is still correct and still honest, but read it as describing a **live
|
||||
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
|
||||
phrased as a phase is part of why the gap went unexamined.
|
||||
|
||||
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
they do not pass**:
|
||||
|
||||
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
|
||||
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
|
||||
## Triggers & branch model (CONTRIBUTING.md)
|
||||
|
||||
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
|
||||
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
|
||||
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
|
||||
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
|
||||
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
|
||||
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
|
||||
|
||||
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
|
||||
## Transition: COMPLETE (2026-09-29)
|
||||
|
||||
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
|
||||
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
|
||||
This section used to describe a migration as pending. It has happened. Gitea is live at
|
||||
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
|
||||
`gate.yml` runs on every PR.
|
||||
|
||||
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
|
||||
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
|
||||
local feature-branch → `main`, the gate enforced by convention + the local runner.
|
||||
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
|
||||
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
|
||||
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
|
||||
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
|
||||
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
|
||||
anyone checking the platform. In reality `main` had **no branch protection** and **zero
|
||||
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
|
||||
between rested on an assertion nobody verified. PR #1 — the first pull request this
|
||||
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
|
||||
|
||||
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
|
||||
Query the API. `infra/gitea/` holds the scripts and the measured state.
|
||||
|
||||
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
|
||||
CI. That is a real coverage gap, not a phase.
|
||||
|
||||
## Configuration (set in Gitea; referenced by name only)
|
||||
|
||||
|
||||
@@ -44,16 +44,31 @@
|
||||
---
|
||||
name: gate
|
||||
|
||||
# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
|
||||
#
|
||||
# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
|
||||
# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
|
||||
#
|
||||
# * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
|
||||
# SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
|
||||
# clock on the one result that actually gates the merge. That cost real time during the
|
||||
# 2026-09-29 plex claim, where the loop is racing a short-lived token.
|
||||
# * The list silently did not match this repo's own convention. Branches here are `feat/**`,
|
||||
# which was absent, so the "fast feedback" never fired for the most common prefix anyway —
|
||||
# a mismatch nobody noticed for a month because its only symptom was a MISSING run.
|
||||
# Any hand-maintained prefix list has that failure mode; deleting it removes the class.
|
||||
# * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
|
||||
# you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
|
||||
# same checks buys nothing.
|
||||
#
|
||||
# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
|
||||
# which no PR context exists.
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- 'feature/**'
|
||||
- 'fix/**'
|
||||
- 'docs/**'
|
||||
- 'chore/**'
|
||||
|
||||
concurrency:
|
||||
group: gate-${{ github.ref }}
|
||||
@@ -95,9 +110,27 @@ jobs:
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
|
||||
# OWN database (branch protection + runner registration) — the blind spot that let `main`
|
||||
# sit unprotected with zero runners for a month while this repo recorded both as working.
|
||||
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
|
||||
#
|
||||
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
|
||||
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
|
||||
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
|
||||
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
|
||||
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
|
||||
# tolerance narrows back to spec-immutability alone.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
|
||||
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
|
||||
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
|
||||
# skipping, this line turns that into a failure and the PR goes red instead of
|
||||
# merging. Green here means the token resolved and the live Gitea matched the
|
||||
# declaration in infra/gitea/expected-branch-protection.json.
|
||||
REQUIRE_GITEA_SETTINGS: "true"
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
|
||||
@@ -11,11 +11,14 @@
|
||||
# v
|
||||
# merge-gate decision [R-TEST-03: touched subset green]
|
||||
#
|
||||
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
|
||||
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
|
||||
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
|
||||
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
|
||||
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
|
||||
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
|
||||
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
|
||||
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
|
||||
# as a plan and never revisited once the plan completed.
|
||||
#
|
||||
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
|
||||
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
|
||||
# statement about the live platform; query the API instead (see infra/gitea/).
|
||||
#
|
||||
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
|
||||
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
|
||||
# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
|
||||
# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
|
||||
# merely left alone. That is a one-way, outward-facing operation against a live authoritative
|
||||
# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
|
||||
# reconcile removes can be restored by hand from this file. It is the same discipline as
|
||||
# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
|
||||
# change rather than to a migration.
|
||||
#
|
||||
# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
|
||||
# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
|
||||
# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
|
||||
# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
|
||||
# devcontainer, which has previously produced false "no records" readings — the API is the only
|
||||
# reading that is both complete and authoritative.
|
||||
#
|
||||
# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
|
||||
# TTL and the proxy flag, not just the value.
|
||||
#
|
||||
# <type>|<name>|<content>|<ttl>|<proxied>
|
||||
#
|
||||
# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
|
||||
# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
|
||||
#
|
||||
# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
|
||||
# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
|
||||
# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
|
||||
# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
|
||||
# cannot touch them regardless of what this branch does:
|
||||
#
|
||||
# *.keithjackson.dev -> 136.51.115.252 legacy stack (the zone wildcard)
|
||||
# brickbreak.keithjackson.dev -> 20.236.242.14 unrelated, off-prem
|
||||
# dev.brickbreak… -> 20.236.242.14 unrelated, off-prem
|
||||
#
|
||||
# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
|
||||
# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
|
||||
# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
|
||||
# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
|
||||
# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
|
||||
# end state for `internal` services and it is also precisely why Pi-hole must already be
|
||||
# answering the LAN authoritatively before this branch merges.
|
||||
#
|
||||
# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
|
||||
# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
|
||||
# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
|
||||
# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
|
||||
# HTTPRoute that already had a published record produced
|
||||
# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
|
||||
# interval (~60s) later, and nothing else in the zone moved.
|
||||
#
|
||||
# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
|
||||
# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
|
||||
# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
|
||||
# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
|
||||
# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
|
||||
# points the registry at a source object that does not exist.
|
||||
A|*.keithjackson.dev|136.51.115.252|1|false
|
||||
A|auth.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
|
||||
A|crafty.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|gitea.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|grafana.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|jackett.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|pihole.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|plex.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|radarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|seerr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|unifi.home.keithjackson.dev|192.168.2.21|1|false
|
||||
TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
|
||||
TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
|
||||
TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
|
||||
TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
|
||||
TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
|
||||
TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
|
||||
TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
|
||||
TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
|
||||
TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
|
||||
TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
|
||||
TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
|
||||
TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
|
||||
TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
|
||||
TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
|
||||
TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
|
||||
TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false
|
||||
@@ -0,0 +1,147 @@
|
||||
# CLAUDE.md — how work lands in this repo
|
||||
|
||||
Read this before doing anything durable. It is the delivery contract for AI coding
|
||||
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
|
||||
Every durable change reaches `main` exactly one way:
|
||||
|
||||
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
|
||||
|
||||
There is no other path. Specifically:
|
||||
|
||||
- **Do not `git push origin main`.**
|
||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||
- **Do not merge your own PR.** A human approves and clicks merge.
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
|
||||
> That wording is **stale** — it predates the PR flow. It describes what Flux does
|
||||
> (main is what the cluster follows), not what you are permitted to do. Honour this
|
||||
> file, not that line. Fixing those runbooks is tracked work.
|
||||
|
||||
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
|
||||
|
||||
Verified against the live Gitea on 2026-09-29:
|
||||
|
||||
| Expected | Actual |
|
||||
|---|---|
|
||||
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
|
||||
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
|
||||
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
|
||||
| Approval required before merge | ✅ `required_approvals: 1` |
|
||||
|
||||
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
true. They are true now because each was measured, and the push rejection was provoked rather
|
||||
than assumed. **Re-measure before relying on any of them** — see §2b.
|
||||
|
||||
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
|
||||
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
|
||||
|
||||
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||
|
||||
## 2b. Never infer platform state from this repository
|
||||
|
||||
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
|
||||
Both describe what someone believed at the time of writing, and both have been
|
||||
confidently wrong for a month at a stretch.
|
||||
|
||||
On 2026-09-29 the log stated the merge gate was a required status check with a runner
|
||||
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
|
||||
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
|
||||
was wrong, and nothing in the tree could have revealed it, because the tree was the source
|
||||
of the error.
|
||||
|
||||
**So, before asserting that any gate, protection, runner, secret, or workload is in a
|
||||
given state: query the thing itself.**
|
||||
|
||||
| Claim about... | Check |
|
||||
|---|---|
|
||||
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
|
||||
| a runner exists | `GET /api/v1/admin/actions/runners` |
|
||||
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
|
||||
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
|
||||
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
|
||||
|
||||
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
|
||||
"unverified" is a legitimate report, and a far better one than a confident guess that
|
||||
matches the docs.
|
||||
|
||||
**A green tick is not proof anything ran.** Read what actually executed.
|
||||
|
||||
## 3. Gate before you push
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
|
||||
Exit codes are a contract — do not flatten them:
|
||||
|
||||
| rc | Meaning |
|
||||
|---|---|
|
||||
| 0 | PASS |
|
||||
| 1 | BLOCKED — something failed |
|
||||
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
|
||||
|
||||
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
|
||||
beside this one; that single skip is expected and tolerated. Any *other* skip is a
|
||||
real gap in your run.
|
||||
|
||||
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
|
||||
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
|
||||
|
||||
### Running the gate from a git worktree
|
||||
|
||||
Two unrelated failures, both with known workarounds — put both in any agent brief:
|
||||
|
||||
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
|
||||
and exists only in the main checkout. Do **not** copy it (it is a credential).
|
||||
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
|
||||
normally.
|
||||
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
|
||||
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
|
||||
the `ENGINE` hook — a shim that intercepts `run` and re-execs
|
||||
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
|
||||
`tracked_files()` is this bug, not a broken tree.
|
||||
|
||||
## 4. Branch naming
|
||||
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
|
||||
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
|
||||
|
||||
## 5. Cluster access
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the
|
||||
pinned container. Always name the context explicitly:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
|
||||
default has already changed once. Never rely on it. Do not run two `dev.sh`
|
||||
invocations concurrently.
|
||||
|
||||
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
|
||||
|
||||
## 6. Secrets
|
||||
|
||||
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
|
||||
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
|
||||
list key names or object names only. A password has been leaked into a transcript
|
||||
this way before.
|
||||
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
|
||||
the single creation-rules file; do not add a second one in a subtree.
|
||||
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
|
||||
flow. Do not design around committing one — read the expiry from the issuing page and
|
||||
treat a burnt token as free to retry.
|
||||
|
||||
## 7. Before you finish
|
||||
|
||||
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
|
||||
verified, and what you did **not** verify. State gaps plainly — an honest
|
||||
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
|
||||
burned by the difference.
|
||||
@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
|
||||
Platform-admins require TOTP MFA.
|
||||
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
|
||||
required status check; runner `incluster-dind` online (ns gitea-actions).
|
||||
> **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
|
||||
> Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
|
||||
> `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
|
||||
> were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
|
||||
> — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
|
||||
> 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
|
||||
> See the 2026-09-29 entries for the root cause and the fix.
|
||||
|
||||
### What changed this session (durable)
|
||||
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
|
||||
@@ -2424,6 +2431,148 @@ container dies with `PermissionError` on every run; my older base had `0400` and
|
||||
- The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm
|
||||
follows git once the knob moves; it does not move it.
|
||||
|
||||
## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
|
||||
|
||||
Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
|
||||
watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
|
||||
Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
|
||||
rather than papered over with a plausible-looking metric name.
|
||||
|
||||
### The direct signal is not available, verified rather than assumed
|
||||
|
||||
The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
|
||||
reasons, both checked read-only against live prod:
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
|
||||
| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
|
||||
| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
|
||||
|
||||
So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
|
||||
and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
|
||||
egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
|
||||
the BPF map. Nothing was invented to fill the hole.
|
||||
|
||||
What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
|
||||
`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
|
||||
three agree right now:
|
||||
|
||||
```
|
||||
node label homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
|
||||
node-exporter node_network_iface_id{device="tun0"} = 129
|
||||
bpf map Egress IP 10.150.192.86 / Egress Ifindex 129 (all 16 rows, 4 vpn-class pods)
|
||||
```
|
||||
|
||||
(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
|
||||
`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
|
||||
debugging the wrong agent.)
|
||||
|
||||
### PromQL cannot express the equality test — so the invariant is counted instead
|
||||
|
||||
The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
|
||||
label-to-number conversion, so `label == value` is **not expressible**, and there is no
|
||||
level-triggered form of this comparison. What is expressible is the same invariant counted over a
|
||||
window: **the published label must change at least as often as tun0 is re-created.**
|
||||
|
||||
```promql
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
```
|
||||
|
||||
`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
|
||||
window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
|
||||
kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
|
||||
series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
|
||||
|
||||
### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
|
||||
|
||||
There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
|
||||
extracted from `kubectl kustomize platform-config` output and run verbatim:
|
||||
|
||||
| state | query | result |
|
||||
|---|---|---|
|
||||
| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
|
||||
| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
|
||||
| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
|
||||
| same expr, window 1h | | `[]` |
|
||||
|
||||
The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
|
||||
earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
|
||||
(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
|
||||
a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
|
||||
than a query: over that same 3h the REAL `egress-uplink` label held two values
|
||||
(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
|
||||
because the label was not exported yet — which is the next point.
|
||||
|
||||
Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
|
||||
over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
|
||||
over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
|
||||
`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
|
||||
|
||||
### Behaviour at the edges
|
||||
|
||||
- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
|
||||
and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
|
||||
timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
|
||||
prod for exactly this reason and had to be retrofitted.
|
||||
- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
|
||||
(its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
|
||||
true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
|
||||
budget — is what keeps a normal reconnect quiet.
|
||||
- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
|
||||
still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
|
||||
minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
|
||||
node-exporter textfile collector route, still open.
|
||||
|
||||
### The second rule exists because the first one fails OPEN
|
||||
|
||||
`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
|
||||
degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
|
||||
`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
|
||||
label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
|
||||
same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
|
||||
kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
|
||||
|
||||
`kube-state-metrics.metricLabelsAllowlist` now reads
|
||||
`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
|
||||
DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
|
||||
|
||||
### Rejected, with reasons
|
||||
|
||||
- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
|
||||
vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
|
||||
was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
|
||||
no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
|
||||
alerting surface, itself dependent on a third-party endpoint) — and it could not have been
|
||||
verified on prod today. It remains the right next step.
|
||||
- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
|
||||
the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
|
||||
mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
|
||||
of which this branch could execute or verify. Left open, now with the reason recorded.
|
||||
- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
|
||||
`VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
|
||||
|
||||
### Left for the operator / not settled here
|
||||
|
||||
1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
|
||||
and the stale branch was exercised with a real flap, but no alert has actually reached
|
||||
Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
|
||||
restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
|
||||
through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
|
||||
proof would come from. Neither was fired at prod unasked.
|
||||
2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
|
||||
Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
|
||||
3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
|
||||
alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
|
||||
acknowledgement, that is the textfile-collector design, not a tweak to this one.
|
||||
|
||||
## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way
|
||||
|
||||
Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with
|
||||
@@ -4183,3 +4332,780 @@ restarting, since a restart may silently restore DHCP and destroy the evidence.
|
||||
The README's dev-era addressing was corrected throughout (VIP `.1.54`→`192.168.2.21`, pool and Gateway,
|
||||
device subnet, node name) along with several stale factual claims, since it doubles as the rebuild
|
||||
runbook.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-18 — VERIFIED: internal services no longer advertise themselves publicly
|
||||
|
||||
The R-DNS-03 disclosure is closed for the `internal` tier. Confirmed at the authority
|
||||
(`betty.ns.cloudflare.com`, not a cache):
|
||||
|
||||
| name | public DNS | via Pi-hole `.23` |
|
||||
|---|---|---|
|
||||
| sonarr, radarr, qbittorrent, jackett, semaphore, pihole | **GONE** | 192.168.2.20 |
|
||||
| plex, gitea, seerr, keycloak, nextcloud | 192.168.2.20 (untouched, as designed) | — |
|
||||
|
||||
So split-horizon now genuinely exists: the names resolve internally and are absent externally, which
|
||||
is what `docs/09-dns-and-ingress.md` specified all along. The internal subnet, the Gateway VIP and six
|
||||
service names are no longer publicly enumerable, and the DNS-rebind primitive is gone for those six.
|
||||
|
||||
**A correction to my own sequencing.** I had insisted this could not land until the operator repointed
|
||||
DHCP, on the grounds that removing public records would break LAN resolution. That was wrong, and
|
||||
measuring it took one command: the SOHO router at 192.168.2.1 returns **nothing** for all six names, so
|
||||
the public records were serving nobody on the LAN. The dependency I asserted did not exist. Landing it
|
||||
early closed a security finding a day sooner than my own plan allowed.
|
||||
|
||||
**Operator clarifications recorded** (theirs, 2026-09-18):
|
||||
1. Pi-hole is **DNS only** — the SOHO router remains the DHCP server. Nothing in the Pi-hole change
|
||||
touches DHCP; it exposes udp/53 and tcp/53 and no DHCP feature is enabled. The remaining action is
|
||||
on the router: hand out `192.168.2.23` as the DNS server, ideally with a public resolver secondary
|
||||
so a Pi-hole outage degrades to "internal names stop resolving" rather than "the LAN loses DNS".
|
||||
2. **Reverse proxy by hostname is the accepted design; a separate IP is overkill.** So no second
|
||||
Gateway and no second LoadBalancer address — the existing Gateway already distinguishes by Host/SNI.
|
||||
Per-service public DNS targeting, if wanted, therefore goes through `DNSEndpoint` objects plus the
|
||||
`crd` source rather than a second Gateway carrying a target annotation.
|
||||
3. Minecraft stays open with no whitelist — accepted.
|
||||
|
||||
**Still open, and it is a fact about the WAN, not a repo decision.** The five `public` services have no
|
||||
working public path at all: `136.51.115.252` returns **HTTP 403** for `gitea.home.keithjackson.dev` and
|
||||
presents a placeholder certificate (`CN=%D`), so nothing reverse-proxies these names from outside today.
|
||||
Either the WAN forwards 80/443 to the Gateway at `192.168.2.20` — in which case their records should
|
||||
point at the WAN address — or those five should be re-declared `internal`, because `exposure: public`
|
||||
currently asserts something untrue.
|
||||
|
||||
**Also fixed:** `apps/vms/unifi/unifi-uos-installer.sops.yaml` had been committed **unencrypted** with
|
||||
both fields as literal `REPLACE-ME`, unlike every other `*.sops.yaml` here. `sops <file>` would have
|
||||
failed with "sops metadata not found", and Flux would have created the Secret containing the
|
||||
placeholder strings. Encrypted in place, round-trip verified. Both fields need setting, not just the
|
||||
password — I had omitted `installer-url` from the operator instructions.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-18 — INCIDENT (mine): I printed the console password to the transcript
|
||||
|
||||
**What happened.** Verifying that the `unifi-uos-installer` Secret had reached the cluster, I ran a
|
||||
`kubectl get secret -o jsonpath` that included `{range .data}{@}{end}`. That dumps the whole `data`
|
||||
map, so the base64 of `console-password` was printed to the session transcript. This is precisely the
|
||||
constraint I have written into every agent brief in this repo — *never print decrypted secrets* — and I
|
||||
broke it myself with a careless jsonpath while checking someone else's work.
|
||||
|
||||
**Exposure, stated precisely rather than minimised or inflated:**
|
||||
- The value reached the session transcript, which I cannot scrub.
|
||||
- It did **not** reach git: `git log --all -S<value>` returns **0 commits**.
|
||||
- The scratch file that captured it (`tasks/*.output`) was shredded; a repo- and scratch-wide grep for
|
||||
the string now returns nothing on disk.
|
||||
|
||||
**Why it is not yet a working credential — the one thing that limits this.** The Secret was created
|
||||
`2026-09-14T07:26:43Z`; the running VMI was created `2026-09-12T20:48:30Z`, and its live volume list is
|
||||
only `rootdisk` and `cloudinit` — **the `UOSDATA` disk is not attached to the running VMI at all**. So
|
||||
the guest has never read the Secret, `install-uos` has never executed, and the `ubuntu` account remains
|
||||
locked. The leaked string authenticates nothing today. It would only become valid when the VMI is
|
||||
recreated against the new spec.
|
||||
|
||||
**Required action, and the ordering matters:** rotate `console-password` in the SOPS Secret **before**
|
||||
recreating the VMI. Then the leaked value never becomes a working password anywhere:
|
||||
|
||||
sops apps/vms/unifi/unifi-uos-installer.sops.yaml # set a NEW console-password
|
||||
git commit && git push # let Flux update the Secret
|
||||
kubectl -n unifi delete vmi unifi # only after the rotation has landed
|
||||
|
||||
**The lesson, generalised.** Reading a Secret to check *existence* never requires reading its
|
||||
*values*. `-o jsonpath='{.metadata.name}'` or listing `.data`'s KEYS answers "did it land?"; dumping
|
||||
`.data` answers a question nobody asked. Every previous secret check in this session used a
|
||||
keys-only or boolean form — this one regressed because I appended a convenience field to an
|
||||
otherwise-safe command without re-reading what it expanded to.
|
||||
|
||||
## 2026-09-18 — Two fail-open holes in the gate itself: nested worktrees, and L1 never running
|
||||
|
||||
Three pieces of work, one theme: the gate was reporting green over things it had not checked.
|
||||
|
||||
### 1. `runbooks/node-memory-swap.md` — the procedure the operator asked for
|
||||
|
||||
Keith has a matched 32 GB pair to install and asked which node needs it. **`k3s-prod-1`**: `DIMM1` is a
|
||||
Micron `8ATF2G64HZ-3G2E1` (16 GB, DDR4-3200, 1Rx8), `DIMM3` a `16ATF2G64HZ-2G6E1` (16 GB, DDR4-2667,
|
||||
2Rx8). Capacity is correct; the pair runs at the slower 2667. Nothing is faulty. `k3s-prod-3` is a
|
||||
matched Crucial pair.
|
||||
|
||||
The runbook front-loads the go/no-go checks because this is the awkward node to lose: 24 Longhorn
|
||||
replicas, one of three etcd members (drain leaves quorum at 2 of 3, no further margin), and its
|
||||
rebuilds are the load that starved etcd once. A BLOCKED drain is treated as the safety property
|
||||
working, never something to force past.
|
||||
|
||||
Three things I had written and then removed, because they were not true:
|
||||
|
||||
- **`EtcdIsVoter` does not exist in this cluster.** I had an assertion built on it; it appears nowhere
|
||||
in the repo. The house check is `curl http://127.0.0.1:2381/metrics | grep etcd_server_has_leader`
|
||||
(`runbooks/node-down.md:93`).
|
||||
- **`node-drain-policy` is not pinned** in `platform/longhorn/helmrelease.yaml`. The runbook has the
|
||||
operator READ the live Setting and set it before draining rather than assuming the chart default —
|
||||
that Setting is the entire safety story for 24 replicas.
|
||||
- I had written "rebuild concurrency is deliberately 1". That is the purge/restore limits, not replica
|
||||
rebuild — the same 5-vs-1 confusion already corrected once in `longhorn-degraded.md`.
|
||||
|
||||
Verification hinges on **`Configured Memory Speed`**, not `Speed`: a matched 3200 pair being clocked
|
||||
down still reports `Speed: 3200 MT/s`, so matched part numbers alone do not prove the swap worked.
|
||||
|
||||
Also recorded: the `node-health` artifacts de-duplicate DIMM part numbers, so they can prove "two
|
||||
different modules" but cannot distinguish one module from two identical ones — verification collects
|
||||
`dmidecode` live. **Node 2's inventory remains stale and self-contradictory** (documented as the only
|
||||
single-DIMM node at 16 GB, yet reporting ~31.1 GiB) and needs re-inventory before anyone claims
|
||||
three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug.
|
||||
|
||||
### 2. The whole-repo L0 checks were auditing 18 copies of the repo
|
||||
|
||||
Six discovery sites walked `ROOT` without pruning dot-directories, so they descended into
|
||||
`.claude/worktrees/agent-*/` — full repo copies from isolated agent runs, gitignored at
|
||||
`.gitignore:43`.
|
||||
|
||||
The visible half was false FAILs: four ServiceMonitors "selecting no Service", nine PrometheusRules
|
||||
"under test but not in the repo", every path a worktree path. The whole-repo gate was red for reasons
|
||||
with nothing to do with authored surface.
|
||||
|
||||
The dangerous half is quieter, and is why all six were fixed rather than only the failing ones: a check
|
||||
asking *"does anything in the repo declare X"* can be satisfied by a worktree copy instead of real
|
||||
source. The same bug that invents failures can manufacture a pass.
|
||||
|
||||
`egress-paths-selected-by-rules` already had the fix and documents hitting it (it once saw 5
|
||||
`CiliumEgressGatewayPolicy` objects instead of the 1 authored). Its dot-directory predicate is now
|
||||
applied everywhere, plus both `find` calls in `platform/monitoring/tests/run.sh`, whose `--list`
|
||||
reported each manifest once per worktree — 21 real pairs where it had been returning them ~19x over.
|
||||
|
||||
### 3. L1 rendering never ran on a requirement-scoped gate — closed, with the incident reproduced
|
||||
|
||||
`ci/l1.sh` has passed `--include-scope whole-repo` since it was written, with a comment saying no L1
|
||||
test declared the scope yet and it was wired ahead of time. Nothing ever declared it, so **the entire
|
||||
L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
|
||||
R-NET-05/06/07.**
|
||||
|
||||
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. The justification is one
|
||||
shared code path — `catalog/render.py` against `schemas/servicedefinition.schema.json` — which
|
||||
requirement-scoped selection cannot see.
|
||||
|
||||
The 2026-09-04 external-dns merge is the proof, reproduced rather than asserted.
|
||||
`ci/touched-requirements.sh` attributes `platform/externaldns/helmrelease.yaml` to R-ENV-02,
|
||||
R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — **none** of the rendering requirements — yet that
|
||||
change added `external-dns.alpha.kubernetes.io/controller: none` to the rendered output of all three
|
||||
goldens. Measured with `--print-selection`:
|
||||
|
||||
| selection for that change | L1 tests selected |
|
||||
|---|---|
|
||||
| requirement-scoped only (the 2026-09-04 behaviour) | **0** |
|
||||
| with `--include-scope whole-repo` (how `ci/l1.sh` invokes it) | **3** |
|
||||
|
||||
### 4. …which exposed two defects in the meta-check that guards all of this
|
||||
|
||||
Declaring the scope at L1 made `whole-repo-invariants-wired` (R-TEST-03) fail, and it was right to
|
||||
change behaviour — but for the wrong reason, and it could not say so:
|
||||
|
||||
- **It was layer-agnostic where the design is layer-bounded.** It compared *every* scope-declaring dir
|
||||
against an `--layer l0` selection, assuming only L0 tests would ever declare the scope — while
|
||||
`ci/l1.sh` was written from the opposite premise, that a whole-repo invariant could appear "at any
|
||||
layer". So it blamed the union for correctly excluding another layer. Now partitioned by layer, with
|
||||
the L1 selection and L1 leakage assertions added — which is what makes the L1 declaration
|
||||
load-bearing instead of decorative.
|
||||
- **Every assertion in it failed silently.** The idiom was `<test>` on one line, `chk $?` on the next,
|
||||
under `set -euo pipefail` — so a false test aborted the script *before* `chk` could print. The real
|
||||
failure I hit emitted 26 `ok` lines, rc=1, and no message. All 12 sites now capture the status
|
||||
(`chk $arc`), proven by forcing `MIN_DECLARED=999`: the failure prints its full diagnostic and the
|
||||
run continues to the remaining assertions.
|
||||
|
||||
A meta-check that cannot report why it failed is worse than no meta-check, because the silence reads
|
||||
as a tooling glitch rather than a finding.
|
||||
|
||||
### Gate
|
||||
|
||||
Offline `l0,l1,meta --include-scope whole-repo` in the pinned toolbox: **25 tests, 0 failures, 0
|
||||
errors, exit 0**. L0 whole-repo went 4 failures → 1 → 0 across the two fixes;
|
||||
`whole-repo-invariants-wired` now runs 35 assertions. Host runs report one skip because `promtool` is
|
||||
not on `$PATH` there — the toolbox run is the real result, and that difference is worth remembering
|
||||
before trusting a host-only gate run.
|
||||
|
||||
### PM HANDOFF
|
||||
|
||||
Open, in the order I would take them:
|
||||
|
||||
1. **The memory swap is a free R-STATE-02 proof.** `node-loss` is Phase 7's last exit item and is still
|
||||
proven only on the decommissioned dev cluster. The swap is a planned, supervised single-node loss on
|
||||
prod — exactly the event R-STATE-02 describes. Capturing evidence during it costs nothing extra;
|
||||
manufacturing the event later costs a second maintenance window.
|
||||
2. **Node 2 DIMM re-inventory** — blocked on the vault become password for `node_health`. Ansible only;
|
||||
never `ansible-inventory --list` (it prints decrypted vault values).
|
||||
3. **Workstation DNS, one sudo command.** `/etc/dnsmasq.d/10-home.keithjackson.dev.conf` contains
|
||||
`local=/home.keithjackson.dev/` plus `address=/.home.keithjackson.dev/192.168.1.53` — the whole zone
|
||||
hardcoded to the decommissioned dev cluster, with forwarding forbidden so no upstream could correct
|
||||
it. That is one root cause for both the sonarr failure and `gitea` not resolving. Pi-hole answers
|
||||
correctly (`gitea` → `192.168.2.20`); only this file is wrong. Correction to an earlier claim of
|
||||
mine: there is no separate `20-prod.*` file — it is a replace of this one file. Router DHCP option
|
||||
6 → `.23` is the fix that makes it unnecessary per-machine.
|
||||
4. **UniFi guest has no IP** — needs console access; `cat /etc/netplan/*.yaml` is the confirming command.
|
||||
5. **Five `public` services have no working public path** — waiting on WAN forwarding, which Keith has
|
||||
said is deliberately not ready.
|
||||
6. **18 stale agent worktrees** on already-merged branches remain on disk. No longer load-bearing for
|
||||
the gate; cleaning them up is cosmetic and should check each for uncommitted work first.
|
||||
7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped
|
||||
paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting
|
||||
instance).
|
||||
|
||||
## 2026-09-19 — Overnight: the backup alert was crying wolf, and 49 app manifests were gated by nothing
|
||||
|
||||
Worked autonomously while the operator slept. One theme again: things that reported green, or red, over
|
||||
something other than what they claimed to measure.
|
||||
|
||||
### 1. VeleroNoRecentBackup was firing CRITICAL over backups that protected everything
|
||||
|
||||
`velero-daily-20260919030041`: **PartiallyFailed, 3079 of 3079 items backed up, 2 errors** — both
|
||||
`PodVolumeBackups` for the emptyDir `plugins` + `scratch` volumes of a
|
||||
`homepage-default-kopia-maintain-job-…` pod. Velero runs transient kopia repository-maintenance Jobs in
|
||||
its own namespace, one per BackupRepository; under `defaultVolumesToFsBackup: true` it tries to
|
||||
fs-backup their volumes, the Job finishes, the pod disappears mid-backup, and the PVB fails. Four such
|
||||
failures had accumulated.
|
||||
|
||||
Nothing real was ever missed. But `PartiallyFailed` does not advance
|
||||
`velero_backup_last_successful_timestamp`, so the alert guarding R-STATE-03 fired nightly over Velero
|
||||
cleaning up after itself. **An alert that cries wolf every night is how a real backup failure gets
|
||||
ignored** — which is precisely the failure mode that alert exists to prevent.
|
||||
|
||||
**First fix was right about the bug and wrong about the layer.** I excluded ns `velero`'s volume data
|
||||
(keeping its resources in schedule 2) and explicitly deferred the general fix — skip emptyDir by type —
|
||||
because Velero's `volumeTypes` policy condition was unproven here and an unparseable `policies.yaml`
|
||||
breaks EVERY backup. I wrote "revisit if the shape shows up elsewhere."
|
||||
|
||||
It showed up elsewhere inside the hour. The verification backup came back PartiallyFailed with 4 errors,
|
||||
all in ns `runbooks`: the nginx Deployment's emptyDir `tmp` + `cache`, failing with
|
||||
`error to expose PVB: error getting client pod …: pods "…" not found`. **Cause worth stating plainly: I
|
||||
caused it** — pushing a runbook edit changed the configMap hash, rolled the Deployment, and the backup
|
||||
lost the pods mid-snapshot. Self-inflicted, not a standing defect. But the mechanism is not
|
||||
self-inflicted: a PVB exposes a volume *through its pod*, so any rollout landing in the backup window
|
||||
fails one, and one failed PVB makes the whole Backup PartiallyFailed. On a GitOps cluster that is
|
||||
routine, not an edge case.
|
||||
|
||||
So the property is what matters, not the namespace: an emptyDir is created empty with its pod and
|
||||
destroyed with it, so backing one up cannot protect anything while attempting it can break the backup
|
||||
that protects everything else.
|
||||
|
||||
**The `volumeTypes` risk was retired by testing, not by hoping.** New policy into a THROWAWAY
|
||||
ConfigMap first, live one untouched, proven on prod (velero 1.18.1) with a control:
|
||||
|
||||
| ns `runbooks` | PodVolumeBackups |
|
||||
|---|---|
|
||||
| control, no policy | **4** (2× `tmp`, 2× `cache`), Completed |
|
||||
| with the policy | **0**, Backup Completed, 0 errors |
|
||||
|
||||
The control is the point — "zero PVBs" alone is equally consistent with nothing having been attempted,
|
||||
which is exactly the empty green I shipped earlier in this session with a 24h baseline built from ten
|
||||
minutes of data.
|
||||
|
||||
Then reverted the namespace exclusion as redundant (`plugins`/`scratch` are emptyDir, verified on a live
|
||||
maintain pod) rather than leaving two overlapping fixes.
|
||||
|
||||
**Proof, full cluster, policy live:** `pre-memswap-202609190525` — **Completed, 3926/3926 items, 0
|
||||
errors, 21/21 PodVolumeBackups Completed, 0 emptyDir volumes backed up.** `VeleroNoRecentBackup`
|
||||
cleared. `VeleroBackupPartialFailure` remains at `increase = 1.008` from the pre-fix failure and
|
||||
self-clears about an hour after it, by design — the rule is `increase(...[1h]) > 0`, which is correct.
|
||||
|
||||
That backup also serves as the pre-maintenance backup the memory-swap runbook requires.
|
||||
|
||||
### 2. The runbook claimed the CNPG primaries sit on node 1. They do not.
|
||||
|
||||
Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity,
|
||||
and **k3s-prod-1 holds a REPLICA of each** (`keycloak-db-2`, `nextcloud-db-2`, `semaphore-db-2`). No
|
||||
primary is on node 1, so draining it forces **no failover at all** — the best state this maintenance
|
||||
could ask for. The original sentence was stale session state. The runbook now prints the command to
|
||||
re-derive it, because primaries move after any restart.
|
||||
|
||||
That also settles `CNPGPrimariesCoLocated`, firing for k3s-prod-2 (both `keycloak-db` and
|
||||
`semaphore-db` primaries). **Do not tidy it before the swap.** Rebalancing means promoting a replica,
|
||||
the only node with spare primary capacity is node 1, so "fixing" the alert would put a primary on the
|
||||
node about to be powered off and convert a clean drain into a two-database failover. Rebalance after
|
||||
node 1 is back and caught up. Warning is now in the runbook.
|
||||
|
||||
### 3. 49 app manifests attributed no requirement, so the merge gate ran no behaviour test for them
|
||||
|
||||
`ci/touched-requirements.sh`'s coarse path map had no arm for `apps/*`. A change adding or editing any
|
||||
workload selected **zero** requirement-scoped tests — so none of the L2 behaviour tests that enforce
|
||||
default-deny, egress class or auth mode ran on exactly the changes that can violate them. Kyverno
|
||||
admission and the L0 egress-path check are real backstops, but admission is post-merge and the gate
|
||||
exists to catch it before.
|
||||
|
||||
Also unmapped: `platform/homepage` and `platform/headlamp` — **the C2 UI**. This was already documented
|
||||
as a known gap in `changed_diff()`'s rule-3 comment ("the ONLY route to R-C2-01/02 was rule 3", i.e.
|
||||
editing the C2 test itself), and had sat there since.
|
||||
|
||||
Twelve arms added: `apps/vms/*` (before `apps/*`, since `case` takes the first match), `apps/*`,
|
||||
`platform/homepage|headlamp`, `platform/kubevirt`, `platform/gateway`, `platform/coredns`,
|
||||
`platform/crafty`, `platform/external-snapshotter|backup`, `runbooks/*`, `platform-config/*`,
|
||||
`platform/kustomization.yaml` and `.sops.yaml`.
|
||||
|
||||
`platform/kustomization.yaml` deserves its own note: it is the assembly list, so **one deleted line
|
||||
disables an entire platform component** while every file it points at stays valid on disk. That is the
|
||||
most consequential single-line edit in the repo and it attributed nothing.
|
||||
|
||||
Measured effect for an `apps/` change: **0 → 13 requirement-scoped tests** (3 L1, 8 L2, 2 L3). Coverage
|
||||
stays 51/51, so the gate's "attributed IDs must be covered" assertion does not false-block.
|
||||
|
||||
Ten new assertions in `ci/tests/touched-requirements.test.sh` (26 total, all passing), including the
|
||||
ORDER property — `apps/vms/*` must win over `apps/*` — since that is a real failure mode of `case` and
|
||||
not merely a matter of both arms existing.
|
||||
|
||||
**A correction on my own numbers.** I first reported "192 unmapped files", then "182". Both were
|
||||
inflated by my audit script, not by the repo: the regex used `[^)]*`, which matched across a newline, so
|
||||
the first arm parsed as `case "$1" in\n catalog/*` and never matched anything — and I separately
|
||||
mis-parsed alternation arms like `.gitea/*|ci/*`, which made `ci/` look unmapped when it is mapped.
|
||||
True remainder after this change is **18 files, all prose or captured artifacts** (`docs/plans/`,
|
||||
root READMEs, `.standup-dns/*.txt`).
|
||||
|
||||
### 4. Finished `feat/egress-gateway-staleness-alert` — it had been backlogged because it could not pass the gate
|
||||
|
||||
Merged the two alerts for the silent egress-gateway failure (`VPNEgressSteeringStale` critical,
|
||||
`VPNEgressUplinkLabelMissing` warning) plus the KSM allowlist entry that makes them non-inert.
|
||||
|
||||
Confirmed first that they are **live on prod, not decorative**: `k3s-prod-3` carries
|
||||
`egress-gateway=true` and `egress-uplink=10.150.192.86-139`, `vpn-egress-steering` has been in place 14
|
||||
days, and the label was exported on **0 series** before this change — so the allowlist entry is the
|
||||
switch that turns the alerts on.
|
||||
|
||||
Three whole-repo checks failed the branch, each a real gap: the runbook the alerts link to did not
|
||||
exist (a firing alert would hand the operator a 404); `alerts.yaml` is an already-covered file, so new
|
||||
rules in it must be unit-tested; and the scrape-target guard broke. Wrote
|
||||
`runbooks/vpn-egress-stale.md` (wired into the generator, four required sections) and
|
||||
`platform/monitoring/tests/vpn-egress.test.yaml` (four promtool fixtures — tests 1 and 2 are the SAME
|
||||
reconnect with and without the label following it, which isolates the mechanism: a label value change
|
||||
retires one series and starts another rather than changing a sample).
|
||||
|
||||
**Two of the three failures were defects in the checks, not the branch:**
|
||||
|
||||
- `alert-metrics-have-scrape-targets` asserted the allowlist matched `nodes=[...egress-gateway]` with a
|
||||
literal closing bracket — that the list held *exactly* that one label. Correctly adding a second label
|
||||
therefore failed the check while the guarded property stayed satisfied. A guard that breaks when you
|
||||
correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding.
|
||||
- `UNTESTED_ALERTS` flagged `VPNTunnelDown` as stale because "asserted on" is a **bare word match over
|
||||
the test file** — my header *comment* mentioning it counted as coverage. I did not delete the register
|
||||
entry to make that go away; instead the fixtures now assert `VPNTunnelDown` **quiet** in all four
|
||||
cases. That required making them faithful: they had no `node_network_receive_bytes_total{device="tun0"}`
|
||||
at all, so `VPNTunnelDown` would have been firing throughout unnoticed, because promtool only checks
|
||||
the alertnames you name. A healthy tun0 passes traffic — and that silence IS the claim these rules
|
||||
exist for.
|
||||
|
||||
**Known, deliberately not fixed:** that word match means any alert named in prose counts as tested.
|
||||
Tightening it to `alertname:` positions could expose other prose-only "coverage", and the register is
|
||||
shrink-only, so the cascade wants doing deliberately — not at 01:00 unattended.
|
||||
|
||||
### 5. Stale worktrees: audited, deliberately not touched
|
||||
|
||||
18 agent worktrees remain under `.claude/worktrees/`. **14 clean, 4 dirty** — the dirty ones being
|
||||
`fix/gate-does-not-pass-on-skipped-stages`, `fix/postbuild-substitution-catches-dollar-mangling`,
|
||||
`fix/scrape-targets-for-inert-alerts` and `fix/qbittorrent-durable-webui-credential`. Every branch is
|
||||
already merged into main, so the uncommitted state is leftover agent scratch rather than unmerged work.
|
||||
|
||||
Left in place. They no longer affect the gate (the scanner now skips dot-directories), so removal is
|
||||
disk hygiene, and deleting someone's uncommitted changes unasked is not a call to make while they are
|
||||
asleep. `git worktree remove` preserves branches and their commits, so only that scratch is at risk.
|
||||
|
||||
### PM HANDOFF
|
||||
|
||||
1. **R-STATE-02 is still the Phase 7 exit item, and it is blocked on permission, not on readiness.**
|
||||
`./conformance/run.sh --layer l3 --requirements R-STATE-02 --record`. The permission classifier
|
||||
refuses it and I did not route around it — hand-rolling the cordon would defeat the guard. I read the
|
||||
test first: it cordons ONE node and evicts only its OWN writer pod, never `kubectl drain`; kubelet
|
||||
and etcd stay up so quorum is untouched; PVC and pods live in the per-test namespace; and `finally`
|
||||
uncordons via a unique label even on failure or panic. Preconditions verified: 3 Ready nodes,
|
||||
`longhorn` default SC, nothing pre-cordoned, 24/24 volumes healthy. Either run it, or add a Bash
|
||||
permission rule.
|
||||
2. **The memory swap is still the cheapest R-STATE-02 evidence** — a supervised single-node loss on prod
|
||||
is the event that requirement describes. Node 1 currently holds no DB primary, which is the ideal
|
||||
window.
|
||||
3. **Node 2 DIMM re-inventory** — blocked on the vault become password. Never `ansible-inventory --list`.
|
||||
4. **Pi-hole needs A records** for `unifi.prod` → `.21` and `crafty-mc.prod` → `.22` before router DHCP
|
||||
option 6 → `.23` is safe. It currently answers `.20` for both, and neither service speaks HTTP
|
||||
through that Gateway VIP, so the LAN-wide change would break UniFi adoption and Minecraft. Until
|
||||
then the per-machine dnsmasq override is load-bearing.
|
||||
5. **UniFi guest has no IP** — needs console; `cat /etc/netplan/*.yaml` is the confirming command.
|
||||
6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
|
||||
7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
|
||||
manifests nothing applies (the keycloak one would stand up a conflicting instance).
|
||||
|
||||
## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.**
|
||||
|
||||
Went to land the plex claim wiring through the documented PR flow and discovered the flow has
|
||||
never run. **PR #1 is the first pull request this repository has ever had.** It is still queued.
|
||||
|
||||
### What is actually true of the live Gitea (verified via API, 2026-09-29)
|
||||
|
||||
| Claimed (DECISIONS-LOG 2026-08-31) | Measured |
|
||||
|---|---|
|
||||
| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` |
|
||||
| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level |
|
||||
|
||||
Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run.
|
||||
PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with
|
||||
nothing to execute them, and will sit there indefinitely.
|
||||
|
||||
This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it
|
||||
was invisible precisely because the log asserted the opposite. Every "the gate is enforcing"
|
||||
statement since 2026-08-31 rested on an unverified claim.
|
||||
|
||||
### Ordering trap — read before fixing
|
||||
|
||||
**Do not enable required status checks until a runner is live.** Requiring
|
||||
`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR
|
||||
that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real
|
||||
PR, *then* turn on enforcement with the context name CI actually reported.
|
||||
|
||||
Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green),
|
||||
but it still publishes a `gate / conformance (pull_request)` context. If that context is ever
|
||||
listed as required, it deadlocks every PR for the same reason. Wire it or exclude it.
|
||||
|
||||
### Delivery contract written down
|
||||
|
||||
Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the
|
||||
branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than
|
||||
the rule (so agents follow the rule rather than what they can get away with), and carries the
|
||||
toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret
|
||||
handling rules.
|
||||
|
||||
It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were
|
||||
accurate before the PR flow and are stale now. Correcting them is tracked, not done.
|
||||
|
||||
### Also found, not fixed
|
||||
|
||||
- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch
|
||||
pushes get no fast-feedback run. PR trigger is unaffected.
|
||||
- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` +
|
||||
`gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a
|
||||
stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator).
|
||||
- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that
|
||||
before spending a token — `runbooks/plex-claim.md` §First commands.
|
||||
- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places,
|
||||
**uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README
|
||||
says nothing about expiry. The number is plausible community folklore, not a measured fact.
|
||||
The authoritative reading is the countdown rendered on plex.tv/claim at issue time.
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **Operator:** diagnose the gitea-actions runner; get one registered.
|
||||
2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only
|
||||
then add the gate as a required status check.
|
||||
3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
|
||||
4. Claim plex, after checking it is not already claimed.
|
||||
5. Correct the 37 runbooks' `git push origin main` wording.
|
||||
|
||||
## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
|
||||
|
||||
Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
|
||||
rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
|
||||
present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
|
||||
library scan is running over the 8.33T NFS share.
|
||||
|
||||
### The finding: both halves of the 2026-08-31 CI claim were false
|
||||
|
||||
That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
|
||||
`incluster-dind` online". Measured against the live API on 2026-09-29:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner online | **zero** runners at repo, org AND site level |
|
||||
|
||||
**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
|
||||
for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
|
||||
statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
|
||||
and the reason it hid so long is that the log itself was the evidence.
|
||||
|
||||
### Root cause of the dead runner
|
||||
|
||||
`act_runner` registers **once** and persists its identity in `/data/.runner` on the
|
||||
`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
|
||||
**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
|
||||
runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
|
||||
genuine re-registration (`.runner is missing` → `Runner registered successfully`).
|
||||
|
||||
It then did not come back for several minutes, because Flux could not recreate the PVC:
|
||||
`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
|
||||
`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
|
||||
`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
|
||||
**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
|
||||
|
||||
### Corrections to things this log and the runbooks asserted
|
||||
|
||||
- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
|
||||
`success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
|
||||
Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
|
||||
than a deadlock, because a deadlock is loud. Never require it; require
|
||||
`gate / gate (pull_request)` alone. Corrected in #3.
|
||||
- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
|
||||
With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
|
||||
which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
|
||||
`infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
|
||||
- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
|
||||
documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
|
||||
issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
|
||||
`catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
|
||||
comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
|
||||
|
||||
### Measured, once the pipeline actually worked
|
||||
|
||||
`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
|
||||
inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
|
||||
That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
|
||||
|
||||
### State now
|
||||
|
||||
- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
|
||||
- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
|
||||
- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
|
||||
instruction had become actively blocked, not merely stale.
|
||||
- `infra/plex/claim-ship.sh` drives the claim loop end to end.
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
|
||||
prerequisite is met: the gate has been observed green on real PRs.
|
||||
2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
|
||||
org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
|
||||
fixed would leave nobody able to approve.)
|
||||
3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
|
||||
bypass. `user_can_push:false` is the API's opinion, not proof.
|
||||
4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
|
||||
5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
|
||||
FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
|
||||
the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
|
||||
6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
|
||||
|
||||
## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
|
||||
|
||||
Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
|
||||
measured against the live API or cluster, not inferred from this repository.
|
||||
|
||||
### Final state
|
||||
|
||||
| | Verified |
|
||||
|---|---|
|
||||
| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
|
||||
| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
|
||||
| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
|
||||
| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
|
||||
| Gate time | ~1m35s |
|
||||
| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
|
||||
|
||||
### What was corrected in the documentation
|
||||
|
||||
`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
|
||||
The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
|
||||
no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
|
||||
as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
|
||||
`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
|
||||
the SOPS MAC, and not to move the rendered golden).
|
||||
|
||||
### A claim of mine that shipped wrong, and is now corrected in place
|
||||
|
||||
PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
|
||||
`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
|
||||
immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
|
||||
landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
|
||||
next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
|
||||
benign startup line would slow every subsequent poll. The manifest comment now says so.
|
||||
|
||||
### The root cause nobody has fixed yet
|
||||
|
||||
Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
|
||||
conformance gate. **No check exists that could have caught "protection off, runner gone."**
|
||||
`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
|
||||
conformance check that reads the live Gitea API and fails on divergence is the highest-value
|
||||
remaining work in this repo, and it is the difference between this having been fixed and this
|
||||
being unable to recur.
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
|
||||
2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
|
||||
CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
|
||||
unverified on every merge.
|
||||
3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
|
||||
4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
|
||||
If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
|
||||
drop back to relying on the status check alone.
|
||||
|
||||
## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
|
||||
|
||||
`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
|
||||
done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
|
||||
admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
|
||||
were bypassable by exactly the account that performs every merge.
|
||||
|
||||
**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
|
||||
merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
|
||||
them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
|
||||
|
||||
Two process failures, both mine, both the same shape as the original defect:
|
||||
|
||||
1. **I verified with the wrong view.** `/branches/main` reports `protected`,
|
||||
`enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
|
||||
showed everything green while the rule was overridable. The authoritative object is
|
||||
`/branch_protections/main` — read it and enumerate its keys.
|
||||
2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
|
||||
`apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
|
||||
false" but means "key absent" — so a PATCH that silently did nothing looked like a real
|
||||
before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
|
||||
`to_entries`.
|
||||
|
||||
Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
|
||||
sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
|
||||
|
||||
**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
|
||||
approved — and that has not been observed yet. This PR is the test: if it merges without an
|
||||
approval, the fix did not work and the finding stands.
|
||||
|
||||
### The governance question this exposes
|
||||
|
||||
There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
|
||||
approvals now genuinely required, an agent-authored PR is approved by the human — which is the
|
||||
intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
|
||||
account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
|
||||
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
|
||||
that actually verifies something; or add a second human account. **This is a decision for the
|
||||
operator, not a default to be chosen silently.**
|
||||
|
||||
## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
|
||||
|
||||
The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
|
||||
claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
|
||||
open is listed below with the reason each is not merged.
|
||||
|
||||
### MERGE THESE — both gated green
|
||||
|
||||
| PR | What | Risk |
|
||||
|---|---|---|
|
||||
| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
|
||||
| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
|
||||
|
||||
(#11, the Gitea drift detector, was merged during this session — hence the action below.)
|
||||
|
||||
Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
|
||||
(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
|
||||
|
||||
### DO THIS FIRST — #11 is merged and its detector is currently INERT
|
||||
|
||||
`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
|
||||
|
||||
./infra/gitea/set-ci-token.sh # mints + stores GITEA_SETTINGS_TOKEN, never prints it
|
||||
|
||||
Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
|
||||
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
|
||||
instead of skipping. Doing that before seeing it green locks the repo.
|
||||
|
||||
Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
|
||||
unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
|
||||
green run, and it is saying so honestly. The token is what turns it green.
|
||||
|
||||
### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
|
||||
|
||||
This is the part that will silently waste an hour otherwise. None of these are in the repo, by
|
||||
design, and without them the toolbox and the cluster are unreachable:
|
||||
|
||||
1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
|
||||
read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
|
||||
workstation; the public recipient in `.sops.yaml` is only half the pair.
|
||||
2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
|
||||
on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
|
||||
here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
|
||||
3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
|
||||
`~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
|
||||
|
||||
### STATE, MEASURED — do not re-derive it from this log, re-measure it
|
||||
|
||||
`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
|
||||
|
||||
protected:true user_can_push:false enable_status_check:true
|
||||
status_check_contexts:["gate / gate (pull_request)"] required_approvals:1
|
||||
block_admin_merge_override:true runner incluster-dind: online gate ~1m35s
|
||||
|
||||
### NEXT WORK, in priority order
|
||||
|
||||
1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
|
||||
boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
|
||||
bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
|
||||
collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
|
||||
**Tighten the admission policy's image pattern to the literal digest when that image exists.**
|
||||
2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
|
||||
only changes which way the stage fails (see #12).
|
||||
3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
|
||||
Actions **variables and secrets** — the next-nearest invisible surface.
|
||||
4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
|
||||
to approve. The operator holds ultimate authority and does not require that review; revisit if
|
||||
it becomes friction.
|
||||
|
||||
### THE LESSON, because it caused every finding here
|
||||
|
||||
Four times this session the repository was treated as evidence about the live platform, and four
|
||||
times it was wrong: the push that would "be rejected" (it would not have been), the token that
|
||||
"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
|
||||
skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
|
||||
fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
|
||||
not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
|
||||
|
||||
**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
|
||||
|
||||
## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
|
||||
|
||||
Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
|
||||
instructions to continue unattended and merge on green CI. Two things came out of that.
|
||||
|
||||
### 1. The detector was INERT, and would have stayed inert silently
|
||||
|
||||
`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
|
||||
prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
|
||||
|
||||
HTTP 400 {"message":"invalid secret name"}
|
||||
|
||||
The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
|
||||
exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
|
||||
have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
|
||||
have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
|
||||
nearly walked into three times.
|
||||
|
||||
Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
|
||||
status AND reads the secret back before claiming success. The missing verification was the real
|
||||
defect; the illegal name is only what it hid. The secret is stored and verified present, and
|
||||
**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
|
||||
matters, because a skip would have made it red.
|
||||
|
||||
The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
|
||||
|
||||
### 2. No agent-authored PR can be merged without the operator — by design
|
||||
|
||||
Merging PR #12 as `gitea-admin` returned:
|
||||
|
||||
HTTP 405 {"message":"Does not have enough approvals"}
|
||||
|
||||
`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
|
||||
(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
|
||||
something an agent can execute here.** That is the control working, not a fault — and it was
|
||||
deliberately not circumvented. Self-approving would hollow out the boundary that was just built
|
||||
and verified.
|
||||
|
||||
**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
|
||||
human approval (the honest reading of the control), or add a second human reviewer account, or
|
||||
lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
|
||||
verifies something. Whichever, change the DECLARATION in
|
||||
`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
|
||||
correctly fail the gate on the drift.
|
||||
|
||||
### Open PRs, all gated green, none mergeable by an agent
|
||||
|
||||
| PR | What |
|
||||
|---|---|
|
||||
| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
|
||||
| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
|
||||
| #14 | This handoff |
|
||||
| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
|
||||
|
||||
Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
|
||||
#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
|
||||
|
||||
### Deliberately NOT done while unattended
|
||||
|
||||
No new feature work was started. Five unmerged PRs already stack review burden, and building the
|
||||
next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
|
||||
cannot land any faster. The next piece is specified and ready to start once these are merged:
|
||||
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
|
||||
boundary landed in #13.
|
||||
@@ -95,8 +95,22 @@ spec:
|
||||
disk:
|
||||
bus: virtio
|
||||
interfaces:
|
||||
# MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
|
||||
# KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
|
||||
# /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
|
||||
# that file while the NoCloud instance-id is unchanged — and the instance-id here is
|
||||
# domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
|
||||
# handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
|
||||
# administratively DOWN with no address and no DHCP attempt.
|
||||
# That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
|
||||
# device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
|
||||
# (because the per-instance modules were skipped), yet UOS is installed and
|
||||
# qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
|
||||
# Pinning the MAC to the value the current VMI already holds makes the guest's netplan
|
||||
# match permanently, so this cannot recur on the next recreation.
|
||||
- name: default
|
||||
masquerade: {}
|
||||
macAddress: a6:d7:c0:b3:72:fa
|
||||
# Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
|
||||
# these and (b) surfaces them as containerPorts on the virt-launcher pod so the
|
||||
# LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
|
||||
|
||||
@@ -5,8 +5,8 @@ metadata:
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
installer-url: ENC[AES256_GCM,data:CwdpCy4dKbJa7A==,iv:zdKrFad7Lg2g7bM7ALLPzp/3VIjXCahZABEscpJCYF4=,tag:inlLeAKrPrt9qvr8/Y4Hiw==,type:str]
|
||||
console-password: ENC[AES256_GCM,data:F+3lMUjA092RVg==,iv:h7NenRZUEJxTjzRDGCLBMHRtLl9h122wLvUvNGfmB50=,tag:Emo5y0qOUBptnXK03sgadQ==,type:str]
|
||||
installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
|
||||
console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -16,14 +16,14 @@ sops:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLMHpXYVhkblBNQWJ2NlZW
|
||||
WXJPSDcxUEdYakN0R1Nydkc1a2E5anJSNEV3CmM1MUpiNzZUa0dKQXNRZGpTRm9l
|
||||
UnMvMTNDL3gvUG1jVFdBbUI3aU01RkkKLS0tIGh2OUNyZzFpMXI1SWFSZENPNVVJ
|
||||
R2V4N3plREx5UGJ2emprTS9WRzljRXMKCWIHr3+L+H2QxocHmA5oDdjZBfvo2Z5F
|
||||
6JLNGs0M7O0X5n+IUqyM+pU+BbqHX+y2hTOSwOgCSV9Zw+ME1kUTKg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
|
||||
WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
|
||||
TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
|
||||
cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
|
||||
kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-14T07:13:55Z"
|
||||
mac: ENC[AES256_GCM,data:A2JQR++tYY9XrRmN1x24klU4xpanx0Ab64AIPGfvtHUxwtG2JLCMMAHIe64eM60FIKT5ibmnVVeuRIUoDwm5d3WpdX2gmGpzxyr9TDhEcEA2ENQGP5TydnmcmchNcu//FfXTXTgm91+WjBke06gibKEF9H+65s2YbkOtheFUjqA=,iv:tQkW7xDGMb8+35QAx+svTQDEwlvW8AQfouopeRqtNTg=,tag:fLJ58ovDbf5qy0sxBqUfPw==,type:str]
|
||||
lastmodified: "2026-09-19T02:47:13Z"
|
||||
mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -268,6 +268,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -17,6 +17,7 @@ resources:
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-bootstrap-secret.sops.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
|
||||
@@ -232,6 +232,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
|
||||
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
|
||||
#
|
||||
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
|
||||
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
|
||||
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
|
||||
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||
# not prose, so the upstream conditions below are described in words.
|
||||
#
|
||||
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
|
||||
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
|
||||
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
|
||||
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
|
||||
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
|
||||
#
|
||||
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
|
||||
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
|
||||
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
|
||||
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
|
||||
#
|
||||
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
|
||||
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
|
||||
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
|
||||
#
|
||||
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||
# the same file extracted from the running `plexinc/pms-docker:latest` image):
|
||||
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
|
||||
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
|
||||
# README: "If server is already logged in, this parameter is ignored."
|
||||
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
|
||||
# empty body. curl's exit status is never checked, the sed for the authentication-token element
|
||||
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
|
||||
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
|
||||
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
|
||||
# It does NOT crashloop.
|
||||
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
|
||||
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
|
||||
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
|
||||
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
|
||||
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
|
||||
# by hand at claim time.
|
||||
#
|
||||
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
|
||||
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||
# Never decrypt this file into a shell, a log, or a shared terminal.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plex-claim
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
type: Opaque
|
||||
stringData:
|
||||
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
|
||||
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
|
||||
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
|
||||
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-29T06:31:16Z"
|
||||
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -132,6 +132,11 @@ spec:
|
||||
value: Etc/UTC
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
|
||||
@@ -344,6 +344,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -4,7 +4,13 @@
|
||||
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
|
||||
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
|
||||
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
|
||||
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
|
||||
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
|
||||
# Gitea API and writes it here via `sops set`, without the value ever being printed.
|
||||
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
|
||||
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
|
||||
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
|
||||
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
|
||||
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
@@ -14,7 +20,7 @@ metadata:
|
||||
app.kubernetes.io/name: act-runner
|
||||
type: Opaque
|
||||
stringData:
|
||||
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
|
||||
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -30,8 +36,8 @@ sops:
|
||||
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
|
||||
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-26T04:48:14Z"
|
||||
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
|
||||
lastmodified: "2026-09-29T05:48:35Z"
|
||||
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+69
-106
@@ -99,70 +99,79 @@ spec:
|
||||
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
||||
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
||||
automountServiceAccountToken: false
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
|
||||
#
|
||||
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
|
||||
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
|
||||
# running alongside them, and is terminated after them. kubelet holds `runner` until
|
||||
# this container's startupProbe passes, so the ordering is enforced by the kubelet
|
||||
# instead of hand-rolled shell in the runner.
|
||||
#
|
||||
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
|
||||
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
|
||||
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
|
||||
# depended on bare dollar-names surviving it untouched. There is now no shell here at
|
||||
# all, so there is nothing to escape or mis-escape.
|
||||
#
|
||||
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
|
||||
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
|
||||
# and only THEN begins listening, so the port can accept a connection while the certs
|
||||
# are absent or incomplete -- and act_runner hard-exits on a missing
|
||||
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
|
||||
# are usable and that the daemon answers.
|
||||
#
|
||||
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
|
||||
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
|
||||
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
|
||||
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
|
||||
# attempt, landing microseconds after dockerd begins listening and cut off by
|
||||
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
|
||||
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
|
||||
# startup line would slow every subsequent poll. Left documented rather than silenced.
|
||||
#
|
||||
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
|
||||
initContainers:
|
||||
- name: dind
|
||||
restartPolicy: Always # <- this field is what makes it a SIDECAR
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
- docker
|
||||
- --host=tcp://127.0.0.1:2376
|
||||
- --tlsverify
|
||||
- --tlscacert=/certs/client/ca.pem
|
||||
- --tlscert=/certs/client/cert.pem
|
||||
- --tlskey=/certs/client/key.pem
|
||||
- info
|
||||
periodSeconds: 1
|
||||
failureThreshold: 60
|
||||
containers:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
|
||||
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
|
||||
# that used to live here are gone: kubelet now holds this container until the dind
|
||||
# sidecar's startupProbe passes (see initContainers above).
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
@@ -198,52 +207,6 @@ spec:
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
|
||||
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -227,6 +227,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
+46
-9
@@ -32,11 +32,46 @@ env:
|
||||
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
||||
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
||||
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
||||
# single API POST) against the running server; the claim then persists on the /config PVC and
|
||||
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
||||
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
|
||||
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
|
||||
# workload). Plex cannot create libraries over the mounted bulk share while the server is
|
||||
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||
# half of that reasoning survived contact with the upstream source.
|
||||
#
|
||||
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
|
||||
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
|
||||
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
|
||||
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
|
||||
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
|
||||
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
|
||||
#
|
||||
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
|
||||
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
|
||||
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
|
||||
# committed value is the spent token from that claim rather than the original placeholder.
|
||||
#
|
||||
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||
# the running `:latest` image), NOT assumed:
|
||||
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
|
||||
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
|
||||
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
|
||||
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
|
||||
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
|
||||
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
|
||||
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
|
||||
# default 0, which continues startup even when a cont-init script fails.
|
||||
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
|
||||
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
|
||||
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
|
||||
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
|
||||
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
||||
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
||||
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
||||
@@ -56,10 +91,12 @@ storage:
|
||||
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
||||
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
||||
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
||||
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
||||
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
|
||||
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
|
||||
# exchange is then skipped on every later start. (The older note here claimed a
|
||||
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
|
||||
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
|
||||
# failure, and that is why one is committed.)
|
||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||
# downloads/ movies/ photos/ tv/).
|
||||
|
||||
+179
-2
@@ -80,6 +80,18 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
|
||||
BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
|
||||
GATEWAY_NAME = "shared"
|
||||
GATEWAY_NS = "gateway"
|
||||
# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
|
||||
# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
|
||||
# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
|
||||
# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
|
||||
# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
|
||||
# typo — "no DNS controller owns this route" — and because the one value that must never be
|
||||
# emitted here is external-dns's own, which a literal `none` cannot drift into.
|
||||
EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
EXTERNALDNS_CONTROLLER_DISOWN = "none"
|
||||
# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
|
||||
# assert the disown value is not it; never emitted.
|
||||
EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
|
||||
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
|
||||
MONITORING_NS = "monitoring" # where Prometheus runs; the only admitted scrape source
|
||||
|
||||
@@ -836,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list:
|
||||
|
||||
|
||||
def _route_meta(s: dict, route: dict) -> dict:
|
||||
"""HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
|
||||
`exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
|
||||
public DNS record, and now cannot)."""
|
||||
meta = {
|
||||
"name": f"{s['name']}-{route['port']}",
|
||||
"namespace": s["name"],
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
}
|
||||
ann = _auth_annotations(s)
|
||||
ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
|
||||
if ann:
|
||||
meta["annotations"] = ann
|
||||
return meta
|
||||
@@ -1039,6 +1054,64 @@ def _auth_annotations(s: dict) -> dict:
|
||||
return {}
|
||||
|
||||
|
||||
def _externaldns_route_annotations(s: dict) -> dict:
|
||||
"""ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
|
||||
|
||||
`internal` renders NO public DNS record. That sentence was in the contract from the start and
|
||||
nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
|
||||
`internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
|
||||
Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
|
||||
(jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
|
||||
subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
|
||||
of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
|
||||
|
||||
THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
|
||||
`registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
|
||||
from memory. external-dns v0.21.0 `source/annotations/processors.go`:
|
||||
|
||||
func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
|
||||
value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
|
||||
if ok && value != ControllerValue { ...; return true }
|
||||
return false
|
||||
}
|
||||
|
||||
with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
|
||||
`ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
|
||||
`Endpoints()`, BEFORE any hostname is resolved:
|
||||
|
||||
if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
||||
|
||||
so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
|
||||
endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
|
||||
so the same annotation scopes the `service` source too if a rendered Service ever grows an
|
||||
external-dns hostname.
|
||||
|
||||
WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
|
||||
with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
|
||||
service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
|
||||
`catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
|
||||
`exposure: internal` to `public` is the only edit that can publish a name, which is exactly
|
||||
the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
|
||||
conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
|
||||
every gate run, including that this function reads `exposure` rather than naming services.
|
||||
|
||||
WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
|
||||
v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
|
||||
source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
|
||||
what is labelled" posture is available and is STRONGER by default. It is not taken here
|
||||
because its blast radius is not this defect's: seven published names come from hand-authored
|
||||
platform/apps manifests that carry no `exposure` field and that this renderer never sees
|
||||
(auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
|
||||
opt-in filter DELETES all seven the moment it is set, and whether each should stay published
|
||||
is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
|
||||
radius of the reported defect. The filter remains the available hardening once those seven
|
||||
are adjudicated — see platform/externaldns/helmrelease.yaml.
|
||||
"""
|
||||
if s["exposure"] == "internal":
|
||||
return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
|
||||
return {}
|
||||
|
||||
|
||||
def r_servicemonitor(s: dict):
|
||||
"""ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
|
||||
|
||||
@@ -1771,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
convention:
|
||||
- `<svc>.yaml` -> wave(svc)
|
||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
|
||||
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
|
||||
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
|
||||
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
|
||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||
@@ -1797,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
w = waves[name]
|
||||
add(w, f"{name}.yaml")
|
||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
|
||||
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
|
||||
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
|
||||
# Placed only when the file exists, so services without one are unaffected.
|
||||
add(w, f"{name}-claim-secret.sops.yaml")
|
||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||
@@ -2097,6 +2179,96 @@ def _selftest() -> int:
|
||||
"a non-exposed service must not render a gateway-ingress allow (least privilege)",
|
||||
)
|
||||
|
||||
# 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
|
||||
# public DNS record and must be INCAPABLE of rendering one: its route carries
|
||||
# external-dns's own controller annotation set to something other than `dns-controller`,
|
||||
# which v0.21.0's gateway-httproute source (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch) uses to skip the route before any hostname is
|
||||
# resolved. `public` must NOT carry it, or the five public names would vanish instead.
|
||||
# Both directions are asserted here because only asserting the internal half would pass
|
||||
# just as happily if the renderer stamped EVERY route and silently unpublished the zone.
|
||||
dns_int_docs = render(
|
||||
{
|
||||
"name": "di",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(
|
||||
len(dns_int_routes) == 1,
|
||||
"an internal exposed service must still render its HTTPRoute (LAN reachability is "
|
||||
"unchanged — only the PUBLIC record is withdrawn)",
|
||||
)
|
||||
if dns_int_routes:
|
||||
iann = dns_int_routes[0]["metadata"].get("annotations", {})
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
|
||||
)
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
|
||||
"the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
|
||||
"the route rather than releasing it — the one value that inverts the guarantee)",
|
||||
)
|
||||
dns_pub_docs = render(
|
||||
{
|
||||
"name": "dp",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "public",
|
||||
"ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "native"},
|
||||
}
|
||||
)
|
||||
dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
|
||||
if dns_pub_routes:
|
||||
check(
|
||||
EXTERNALDNS_CONTROLLER_KEY
|
||||
not in dns_pub_routes[0]["metadata"].get("annotations", {}),
|
||||
"a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
|
||||
)
|
||||
# The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
|
||||
# ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
|
||||
# field is the likeliest way a new service arrives, and it is the case a per-service
|
||||
# allow-list would miss.
|
||||
dns_dflt_docs = render(
|
||||
{
|
||||
"name": "dd",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
|
||||
== EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
|
||||
"unpublishable (default posture, renderer-contract)",
|
||||
)
|
||||
# The scoping annotation must COEXIST with the auth hints rather than replace them: both are
|
||||
# merged into one `annotations` map, and a forward-auth internal app is the overlap case.
|
||||
dns_fa_docs = render(
|
||||
{
|
||||
"name": "df",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
|
||||
}
|
||||
)
|
||||
dns_fa_ann = next(
|
||||
d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
|
||||
)["metadata"]["annotations"]
|
||||
check(
|
||||
dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
|
||||
and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
|
||||
"DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
|
||||
)
|
||||
|
||||
# 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
|
||||
# PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
|
||||
# declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
|
||||
@@ -2385,7 +2557,8 @@ def _selftest() -> int:
|
||||
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
||||
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
||||
present = {
|
||||
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"a.yaml", "a-claim-secret.sops.yaml",
|
||||
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
||||
"svc-db-backup-credentials-prod.sops.yaml",
|
||||
"gitea.yaml",
|
||||
@@ -2396,6 +2569,10 @@ def _selftest() -> int:
|
||||
plan = plan_apps_waves(plan_sds, present)
|
||||
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||
check(flat.get("a-claim-secret.sops.yaml") == 0,
|
||||
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
|
||||
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
|
||||
"CreateContainerConfigError")
|
||||
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
||||
|
||||
@@ -32,6 +32,9 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
|
||||
l1: ## L1 rendering checks (no cluster; live now)
|
||||
@"$(CI_DIR)l1.sh"
|
||||
|
||||
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
||||
@./gitea-settings.sh
|
||||
|
||||
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
|
||||
@"$(CI_DIR)conformance.sh"
|
||||
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# `ci/` — portable merge gate (runnable today, without Gitea)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
||||
|
||||
The same merge gate the Gitea Actions workflow runs
|
||||
|
||||
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
||||
#
|
||||
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
||||
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
||||
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
||||
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
||||
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
||||
#
|
||||
# MEASURED 2026-09-29, before this guard existed:
|
||||
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
||||
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
||||
# "[ ok ] L2/L3 conformance passed for touched subset"
|
||||
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
||||
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
||||
# it because nobody had ever set the variable.
|
||||
#
|
||||
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
||||
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
||||
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
||||
# makes the stage itself honour that instead of reporting a pass.
|
||||
JUNIT="${CONF_ART}/report.junit.xml"
|
||||
if [ -f "$JUNIT" ]; then
|
||||
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
||||
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
||||
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
||||
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
||||
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
||||
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
||||
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
||||
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
||||
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
||||
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
||||
exit 1
|
||||
fi
|
||||
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
||||
fi
|
||||
|
||||
ok "L2/L3 conformance passed for touched subset"
|
||||
summary "- conformance-l2-l3: **passed**"
|
||||
+26
@@ -112,6 +112,7 @@ run_stage lint "${CI_DIR}/lint.sh"
|
||||
run_stage l0 "${CI_DIR}/l0.sh"
|
||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||
run_stage l1 "${CI_DIR}/l1.sh"
|
||||
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
||||
run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
|
||||
echo >&2
|
||||
@@ -188,6 +189,31 @@ case "${OUTCOME[spec-immutability]}" in
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
||||
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
||||
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
||||
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
|
||||
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
|
||||
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
|
||||
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
|
||||
# token is wired, and the skip becomes a block as it should.
|
||||
case "${OUTCOME[gitea-settings]}" in
|
||||
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
|
||||
skip)
|
||||
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
|
||||
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
|
||||
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
|
||||
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
|
||||
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
|
||||
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
|
||||
NOT_ESTABLISHED+=("gitea-settings")
|
||||
fi ;;
|
||||
pass) ok "gitea-settings: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
|
||||
Executable
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
|
||||
# match what git declares.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS AT ALL
|
||||
# ----------------------------
|
||||
# Every other check in this repo reads the repository, the rendered manifests, or the
|
||||
# cluster. NONE of them can see Gitea's own database — and that is where branch
|
||||
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
|
||||
#
|
||||
# * `main` had NO branch protection, and
|
||||
# * ZERO Actions runners were registered,
|
||||
#
|
||||
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
|
||||
# and human in the loop repeated that claim because the repository said so. PR #1 — the
|
||||
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
|
||||
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
|
||||
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
|
||||
# merged with zero approvals recorded against `required_approvals: 1`.
|
||||
#
|
||||
# Three separate failures, one shape: a setting that READ as enforced while not being, with
|
||||
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
|
||||
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
|
||||
#
|
||||
# WHAT IT ASSERTS
|
||||
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
|
||||
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
|
||||
# 2. at least `.runners_online_min` Actions runners report status "online".
|
||||
#
|
||||
# (2) is not decoration. A required status check with no runner is the deadlock this repo
|
||||
# nearly walked into: every PR pends forever, including the one that would fix it.
|
||||
#
|
||||
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
|
||||
# `protected`, `enable_status_check` and `required_approvals` but NOT
|
||||
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
|
||||
# only account that merges. This stage queries /branch_protections/ deliberately.
|
||||
#
|
||||
# EXIT CODES (the gate contract)
|
||||
# 0 live Gitea matches the declaration
|
||||
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
|
||||
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
|
||||
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
|
||||
# that an unwired credential cannot lock the repository.
|
||||
#
|
||||
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
|
||||
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
|
||||
# the drift it exists to catch is still undetected.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: Gitea platform settings match the declaration"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
|
||||
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
|
||||
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
|
||||
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
|
||||
# it looked like it had worked. A token was minted, no secret existed, and this stage would
|
||||
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
|
||||
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool curl
|
||||
|
||||
if [ -z "$TOKEN" ]; then
|
||||
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
|
||||
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
|
||||
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
|
||||
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
BRANCH="$(jq -r '.branch' "$DECL")"
|
||||
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
|
||||
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
|
||||
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
|
||||
skip " lacks permission). Not treating this as a pass."
|
||||
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
RC=0
|
||||
|
||||
# --- 1. every declared protection key must match the live rule, exactly ---------------
|
||||
while IFS= read -r key; do
|
||||
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
|
||||
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
|
||||
if [ "$want" = "$got" ]; then
|
||||
ok " ${key} = ${got}"
|
||||
else
|
||||
err " ${key}: DECLARED ${want} but LIVE is ${got}"
|
||||
RC=1
|
||||
fi
|
||||
done < <(jq -r '.protection | keys[]' "$DECL")
|
||||
|
||||
# --- 2. a runner must actually be online ----------------------------------------------
|
||||
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
|
||||
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
|
||||
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
|
||||
if [ "${N:-0}" -ge "$MIN" ]; then
|
||||
ok " runners online: ${N} (>= ${MIN})"
|
||||
else
|
||||
err " runners online: ${N}, DECLARED minimum ${MIN}."
|
||||
err " A required status check with no runner leaves every PR pending forever —"
|
||||
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
|
||||
RC=1
|
||||
fi
|
||||
|
||||
if [ "$RC" -eq 0 ]; then
|
||||
ok "Gitea settings match the declaration."
|
||||
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
|
||||
else
|
||||
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
|
||||
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
|
||||
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
|
||||
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
|
||||
fi
|
||||
exit "$RC"
|
||||
@@ -79,12 +79,37 @@ else
|
||||
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
||||
find "$l0_render" -type f -name '*.yaml' -print0 \
|
||||
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
||||
if kyverno test "$l0_render"; then
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
|
||||
# error, bad match block, malformed JMESPath) does not fail the run: every result row
|
||||
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
|
||||
#
|
||||
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
|
||||
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
|
||||
# rows came back `Skip / Invalid Policy`, and the summary read:
|
||||
# Test Summary: 9 tests passed and 0 tests failed
|
||||
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
|
||||
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
|
||||
#
|
||||
# This is not specific to that policy: a typo in restrict-host-namespaces or
|
||||
# secret-provenance would silently disable it and the gate would stay green. Scan the
|
||||
# output for the load failure the exit code hides.
|
||||
l0_kout="$(mktemp)"
|
||||
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
|
||||
if grep -q 'Invalid Policy' "$l0_kout"; then
|
||||
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
|
||||
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
|
||||
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
|
||||
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
|
||||
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
|
||||
rc=1
|
||||
else
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
fi
|
||||
else
|
||||
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
||||
fi
|
||||
rm -f "$l0_kout"
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
|
||||
@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
# --include-scope: union the touched subset with every L1 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
|
||||
# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
|
||||
# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
|
||||
# "someone must remember" is the failure mode this whole change removes; see
|
||||
# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
|
||||
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
|
||||
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
|
||||
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
|
||||
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
|
||||
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
|
||||
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
l1_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
|
||||
@@ -136,6 +136,65 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
||||
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
||||
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
|
||||
#
|
||||
# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
|
||||
# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
|
||||
# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
|
||||
# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
|
||||
# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
|
||||
# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
|
||||
#
|
||||
# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
|
||||
# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
|
||||
#
|
||||
# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
|
||||
# from the PATH and not from the content scan.
|
||||
|
||||
# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
|
||||
# under test here, not just the presence of both arms.
|
||||
git -C "$TMP" checkout -q -b feature-apps-vms main
|
||||
mkdir -p "$TMP/apps/vms/demo"
|
||||
printf 'kind: VirtualMachine\nmetadata:\n name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
|
||||
OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
|
||||
assert_present "R-PLAT-03" "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
|
||||
assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
|
||||
assert_absent "R-PLAT-02" "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
|
||||
|
||||
# A plain ServiceDefinition under apps/ gets the container-workload set.
|
||||
git -C "$TMP" checkout -q -b feature-apps main
|
||||
mkdir -p "$TMP/apps/wave-1"
|
||||
printf 'kind: ServiceDefinition\nmetadata:\n name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
|
||||
OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
|
||||
assert_present "R-PLAT-02" "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
|
||||
assert_present "R-NET-01" "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
|
||||
assert_present "R-NET-06" "$OUT_APPS" "apps/* attributes the egress-class requirements"
|
||||
assert_present "R-IDP-02" "$OUT_APPS" "apps/* attributes the auth-mode requirements"
|
||||
|
||||
# homepage / headlamp ARE the command-centre surface.
|
||||
git -C "$TMP" checkout -q -b feature-c2 main
|
||||
mkdir -p "$TMP/platform/headlamp"
|
||||
printf 'kind: Deployment\nmetadata:\n name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
|
||||
OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
|
||||
assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
|
||||
assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
|
||||
|
||||
# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
|
||||
git -C "$TMP" checkout -q -b feature-assembly main
|
||||
mkdir -p "$TMP/platform"
|
||||
printf 'resources:\n - cilium\n - longhorn\n' > "$TMP/platform/kustomization.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
|
||||
OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
|
||||
assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
|
||||
#
|
||||
|
||||
@@ -177,6 +177,23 @@ path_requirements() {
|
||||
case "$1" in
|
||||
catalog/*) echo "R-GITOPS-02" ;;
|
||||
schemas/*) echo "R-GITOPS-02" ;;
|
||||
# apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
|
||||
# manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
|
||||
# adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
|
||||
# checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
|
||||
# class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
|
||||
# violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
|
||||
# catch it at apply time" is post-merge, and the gate exists to catch it before.
|
||||
#
|
||||
# ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
|
||||
# apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
|
||||
# subject to default-deny (R-NET-01).
|
||||
apps/vms/*) echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
|
||||
# Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
|
||||
# workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
|
||||
# (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
|
||||
# default-deny (R-NET-01).
|
||||
apps/*) echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
|
||||
platform/kyverno/*) echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
|
||||
platform/cilium/*) echo "R-NET-01 R-NET-02 R-NET-05" ;;
|
||||
platform/longhorn/*) echo "R-STATE-01 R-STATE-02" ;;
|
||||
@@ -208,6 +225,39 @@ path_requirements() {
|
||||
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
# The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
|
||||
# rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
|
||||
# itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
|
||||
# docs/12-command-center-ui.md), so editing them must select the C2 tests.
|
||||
platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
|
||||
# KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
|
||||
platform/kubevirt/*) echo "R-PLAT-03 R-OBS-01" ;;
|
||||
# The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
|
||||
# TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
|
||||
platform/gateway/*) echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
|
||||
# CoreDNS is cluster DNS resolution (R-DNS-01).
|
||||
platform/coredns/*) echo "R-DNS-01" ;;
|
||||
# crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
|
||||
# its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
|
||||
platform/crafty/*) echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
|
||||
# Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
|
||||
platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
|
||||
# runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
|
||||
# resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
|
||||
# is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
|
||||
# runbook change is an observability change.
|
||||
runbooks/*) echo "R-OBS-04" ;;
|
||||
# platform-config/ is the env-specific configuration Flux substitutes into and applies.
|
||||
platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
|
||||
# platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
|
||||
# Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
|
||||
# file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
|
||||
# change. That is the most consequential single-line edit in the repo and it attributed nothing.
|
||||
platform/kustomization.yaml) echo "R-GITOPS-01" ;;
|
||||
# .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
|
||||
# here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
|
||||
.sops.yaml) echo "R-SEC-01 R-GITOPS-01" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
|
||||
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# clusters/
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
||||
|
||||
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
|
||||
#
|
||||
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
|
||||
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
|
||||
# meant to verify is worse than no harness.
|
||||
#
|
||||
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
|
||||
# waves needs it.
|
||||
#
|
||||
# NOTE: this reconciles RBAC that is INERT until
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
|
||||
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
|
||||
# trigger identity can exist while the policy bounding it does not.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: conformance
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./conformance/incluster
|
||||
@@ -33,6 +33,7 @@ resources:
|
||||
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
||||
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
||||
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
||||
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
|
||||
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
||||
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
||||
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
||||
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
||||
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# In-cluster conformance Job (L2/L3)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
||||
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
||||
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
|
||||
#
|
||||
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
|
||||
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
|
||||
# reconciling it would create a permanent Job object and fight the trigger for ownership.
|
||||
#
|
||||
# The identities here are inert without
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
|
||||
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
|
||||
# half — see the header of rbac.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- rbac.yaml
|
||||
@@ -174,3 +174,68 @@ subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-runner
|
||||
namespace: conformance
|
||||
---
|
||||
# =============================================================================
|
||||
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
|
||||
#
|
||||
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
|
||||
# that can `create jobs` in this namespace can create a Job with
|
||||
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
|
||||
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
|
||||
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
|
||||
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
|
||||
# access") while appearing to satisfy it.
|
||||
#
|
||||
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
|
||||
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
|
||||
# the necessary half; that policy is the sufficient half. Neither works alone — do not
|
||||
# grant this Role without that policy reconciled and ENFORCING.
|
||||
#
|
||||
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
rules:
|
||||
# Create the run, and watch it to completion.
|
||||
- apiGroups: [batch]
|
||||
resources: [jobs]
|
||||
verbs: [create, get, list, watch]
|
||||
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
|
||||
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
|
||||
# them from the log, so no execution channel into the runner pod is ever granted.
|
||||
- apiGroups: [""]
|
||||
resources: [pods, pods/log]
|
||||
verbs: [get, list, watch]
|
||||
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
|
||||
# failed run), no secrets, no configmaps, no exec, no create on pods.
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: conformance-trigger
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
@@ -0,0 +1,6 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
|
||||
Install the pinned binary or run via the in-cluster Job (incluster/).
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="115.855688" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,232 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 05:22:26 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1789795346371286043
|
||||
victim node (currently hosts the attached volume): k3s-prod-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-prod-2 labeled
|
||||
node/k3s-prod-2 cordoned
|
||||
cordoned + labelled victim node: k3s-prod-2
|
||||
pod "writer" deleted from chainsaw-advanced-boxer namespace
|
||||
evicted writer off the victim node; volume detached from k3s-prod-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
|
||||
recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-prod-2 uncordoned
|
||||
node/k3s-prod-2 unlabeled
|
||||
restored node to service: k3s-prod-2
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-advanced-boxer namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:24:21 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (115.86s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -5,3 +5,4 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}
|
||||
@@ -0,0 +1,42 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
|
||||
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
|
||||
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
|
||||
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
|
||||
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
|
||||
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
|
||||
#
|
||||
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
|
||||
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
|
||||
# would have skipped this check on the very commit that introduced the defect.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: admission-policies-actually-applied
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
|
||||
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
|
||||
exemption that is genuinely not applied — so a policy cannot be authored, tested,
|
||||
green, and absent from the cluster.
|
||||
spec:
|
||||
steps:
|
||||
- name: every-authored-policy-is-applied-or-documented-ci-only
|
||||
description: >-
|
||||
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
|
||||
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — admission-policies-actually-applied
|
||||
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
|
||||
#
|
||||
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
|
||||
#
|
||||
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
|
||||
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
|
||||
# "kept as reference; THIS file is the buildable render root".
|
||||
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
|
||||
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
|
||||
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
|
||||
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
|
||||
#
|
||||
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
|
||||
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
|
||||
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
|
||||
# `platform` Ready at the new revision — and the policy was never in any applied set. The
|
||||
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
|
||||
# which is precisely the state that identity's own RBAC header says must never exist.
|
||||
#
|
||||
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
|
||||
# DEPLOYED. This check asserts the second thing.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
|
||||
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
|
||||
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
|
||||
# policy that IS applied, so the exemption list cannot rot into a lie either way.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import sys, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
|
||||
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
|
||||
# that silently starts being applied is as much a lie as one that silently stops.
|
||||
CI_ONLY = {
|
||||
"disallow-plaintext-secrets":
|
||||
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
|
||||
"admission because Flux decrypts before the Secret reaches the API server. Its "
|
||||
"in-cluster counterpart is admission/secret-provenance.yaml.",
|
||||
}
|
||||
|
||||
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
|
||||
"platform/kyverno/*.yaml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
|
||||
|
||||
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
|
||||
policies = {}
|
||||
for f in files:
|
||||
try:
|
||||
txt = f.read_text()
|
||||
except Exception:
|
||||
continue
|
||||
if "kind: ClusterPolicy" not in txt:
|
||||
continue
|
||||
for line in txt.splitlines():
|
||||
st = line.strip()
|
||||
if st.startswith("name:") and " name:" in line:
|
||||
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
|
||||
break
|
||||
|
||||
check(len(policies) >= 6,
|
||||
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
|
||||
f"too few means the scan broke, and every assertion below would be vacuous")
|
||||
|
||||
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
|
||||
|
||||
unapplied, wrongly_exempt = [], []
|
||||
for rel, name in sorted(policies.items()):
|
||||
# platform-config references these as ../platform/kyverno/...
|
||||
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
|
||||
if name in CI_ONLY:
|
||||
if referenced:
|
||||
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
|
||||
elif not referenced:
|
||||
unapplied.append(f"{name} ({rel})")
|
||||
|
||||
check(not unapplied,
|
||||
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
|
||||
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
|
||||
|
||||
check(not wrongly_exempt,
|
||||
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
|
||||
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
|
||||
|
||||
for name, reason in CI_ONLY.items():
|
||||
check(bool(reason.strip()),
|
||||
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
|
||||
f"list stops being reviewable")
|
||||
|
||||
check(CHECKS >= 4,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
|
||||
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
|
||||
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
|
||||
f"not that they are correct — `kyverno test` is what proves correctness.")
|
||||
PY
|
||||
@@ -256,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"metricLabelsAllowlist:",
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
|
||||
# The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
|
||||
# end in a literal `]`, asserting the list held exactly this one label — so adding a
|
||||
# second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
|
||||
# while the property it guards stayed perfectly satisfied. A guard that breaks when you
|
||||
# correctly extend the thing it guards teaches people to edit the guard, which is how it
|
||||
# stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
|
||||
@@ -448,9 +454,14 @@ def is_golden_fixture(p: Path) -> bool:
|
||||
return "conformance" in parts and "golden" in parts
|
||||
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
@@ -619,7 +630,7 @@ def yaml_docs(path: Path) -> list[dict]:
|
||||
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
|
||||
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts:
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
|
||||
continue
|
||||
txt = read(p)
|
||||
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
|
||||
|
||||
@@ -179,6 +179,10 @@ UNTESTED_FILES = {
|
||||
# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
|
||||
# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
|
||||
# detectors are no longer unasserted.
|
||||
# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
|
||||
# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
|
||||
# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
|
||||
# had to become an assertion rather than a sentence in a comment.
|
||||
UNTESTED_ALERTS = {
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
||||
@@ -195,7 +199,6 @@ UNTESTED_ALERTS = {
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
||||
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
||||
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
||||
@@ -280,11 +283,16 @@ FETCH_RE = re.compile(
|
||||
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
|
||||
# version, the network and the machine are decided silently at test time, and a missing
|
||||
# toolchain becomes a green run instead of an exit 3.
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
|
||||
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
|
||||
for p in sorted(ROOT.rglob("*.sh")):
|
||||
rel = str(p.relative_to(ROOT))
|
||||
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
|
||||
if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
|
||||
continue
|
||||
try:
|
||||
text = p.read_text()
|
||||
@@ -319,7 +327,7 @@ def tracked_yaml() -> list[Path]:
|
||||
return [ROOT / p for p in out.stdout.split("\0") if p]
|
||||
# No usable index (the gate container mounts a worktree whose .git points outside the
|
||||
# mount). Fall back to a filesystem walk rather than auditing nothing.
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
|
||||
|
||||
|
||||
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
|
||||
@@ -473,7 +481,34 @@ for rel in sorted(alert_files):
|
||||
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
|
||||
f"shrink-only: delete the line so the next missing test is caught")
|
||||
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
|
||||
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
|
||||
# "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
|
||||
# This used to be a bare word match over the whole test file, so an alert named anywhere —
|
||||
# including in a PROSE COMMENT — counted as tested. Hit for real while adding
|
||||
# platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
|
||||
# stays quiet during an egress-steering failure, and that sentence alone was enough to make this
|
||||
# check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
|
||||
# the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
|
||||
# outcome this check exists to prevent, produced BY the check.
|
||||
# Now an alert counts only when a test file names it in an `alertname:` position, with comment
|
||||
# lines stripped first so a commented-out assertion does not count either.
|
||||
# MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
|
||||
# asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
|
||||
# registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
|
||||
# mattered because the registers are shrink-only: had any alert been prose-only, tightening would
|
||||
# have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
|
||||
# it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
|
||||
text_no_comments = "\n".join(
|
||||
l for l in text.split("\n") if not l.lstrip().startswith("#")
|
||||
)
|
||||
# BOTH promtool idioms count, because both are real assertions:
|
||||
# alert_rule_test: - alertname: CNPGPrimariesCoLocated
|
||||
# promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
# Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
|
||||
# platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
|
||||
# form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
|
||||
# already better covered than most.
|
||||
covered = [n for n in names
|
||||
if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
|
||||
covered_total += len(covered)
|
||||
for n in names:
|
||||
if n in covered:
|
||||
|
||||
@@ -126,9 +126,14 @@ def read(p: Path) -> str:
|
||||
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
|
||||
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
|
||||
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
|
||||
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
|
||||
"runbooks", "clusters", "bootstrap", "catalog")
|
||||
|
||||
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
|
||||
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
|
||||
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
|
||||
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
|
||||
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
|
||||
# routes and make (8) demand the operator's dashboard link them.
|
||||
#
|
||||
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
|
||||
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
|
||||
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
|
||||
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
|
||||
# become a real delivery root.
|
||||
NON_SERVING_RECONCILED = ("conformance/incluster",)
|
||||
|
||||
|
||||
def tracked_yaml() -> list[Path]:
|
||||
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
|
||||
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
|
||||
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
|
||||
# a path of them would fail on a file that is correct.
|
||||
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
|
||||
def _reconciled_path(d):
|
||||
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
|
||||
|
||||
|
||||
unscanned = sorted({
|
||||
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
|
||||
for p, d in reconcilers
|
||||
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
|
||||
not in DELIVERY_ROOTS
|
||||
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
|
||||
and _reconciled_path(d) not in NON_SERVING_RECONCILED
|
||||
})
|
||||
|
||||
# The exemption's own guard: an exempted tree must actually serve nothing.
|
||||
exempt_serving = sorted({
|
||||
f"{p.relative_to(ROOT)} ({d.get('kind')})"
|
||||
for p, d in docs_all()
|
||||
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
|
||||
and (d.get("kind") == "HTTPRoute"
|
||||
or (d.get("kind") == "Service"
|
||||
and any("hostname" in str(k).lower()
|
||||
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
|
||||
})
|
||||
check(not exempt_serving,
|
||||
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
|
||||
f"gains a route would hide a service from this check entirely "
|
||||
f"{exempt_serving if exempt_serving else ''}")
|
||||
check(len(reconcilers) >= 10 and not unscanned,
|
||||
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
|
||||
f"reconciled tree this check does not read is a service it cannot require a link for "
|
||||
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
||||
# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
|
||||
# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
|
||||
# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
|
||||
# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
|
||||
# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
|
||||
# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
|
||||
# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
|
||||
# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
|
||||
# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
||||
#
|
||||
# Three things were wrong at once, and none of them was visible from inside the cluster:
|
||||
# * the internal subnet, the Gateway VIP and a complete service inventory were publicly
|
||||
# enumerable by anyone who could guess or brute-force a label;
|
||||
# * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
|
||||
# SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
|
||||
# overrides to reach their own services;
|
||||
# * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
|
||||
# and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
|
||||
# respect.
|
||||
#
|
||||
# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
|
||||
# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
|
||||
# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
|
||||
# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
|
||||
# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
|
||||
# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
|
||||
# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
|
||||
# prose. This test makes the DNS guarantee a gate condition instead.
|
||||
#
|
||||
# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
|
||||
# Service this repo authors,
|
||||
# WHEN each catalog service's rendered routes are resolved in the applied tree (apps/),
|
||||
# THEN (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
|
||||
# annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
|
||||
# uses to skip the source object outright (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch, checked before any hostname is resolved), so the
|
||||
# route cannot become an endpoint for ANY name it declares;
|
||||
# (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
|
||||
# renderer that stamped every route would satisfy (1) perfectly while silently
|
||||
# withdrawing the five public services' records, and `policy: sync` would delete them;
|
||||
# (3) the same holds for the `service` source: an internal service's rendered Service may
|
||||
# not carry an external-dns hostname annotation without the disown annotation beside it.
|
||||
# Two sources are configured, so closing only the route half is half a guarantee;
|
||||
# (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
|
||||
# the emitting function must read `exposure`, must not name any catalog service, and
|
||||
# must not be able to emit external-dns's own `dns-controller` value (the one value that
|
||||
# CLAIMS a route rather than releasing it, inverting the guarantee while looking
|
||||
# plausible). A per-service list is how twelve ServiceMonitors acquired one identical
|
||||
# wrong port name in a single edit;
|
||||
# (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
|
||||
# SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
|
||||
# annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
|
||||
# yield it (the contract's default posture is `internal`, and omitting the field is the
|
||||
# likeliest way a new service arrives). Without this the check would still pass if the
|
||||
# renderer stopped working and someone hand-maintained apps/;
|
||||
# (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
|
||||
# DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
|
||||
# scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
|
||||
# renderer to read, and they are the seven names that REMAIN publicly advertised with an
|
||||
# RFC1918 answer after this change. The register is shrink-only: an entry that stops
|
||||
# matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
|
||||
# quietly grow, and adding a new one is a review event;
|
||||
# AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
|
||||
# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
|
||||
# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
|
||||
# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
|
||||
# Same reasoning as its whole-repo siblings.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
|
||||
# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
|
||||
# and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
|
||||
# - That the record is actually GONE from Cloudflare. That is a live read against an external
|
||||
# authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
|
||||
# the positive direction (a public name DOES appear). The negative direction was proven
|
||||
# MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
|
||||
# carrying the annotation produced no record at all, and adding the annotation to a route that
|
||||
# already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
|
||||
# entry within one ~60s reconcile — but a manual proof is not a gate.
|
||||
# - That an `internal` name is unreachable from the internet. No DNS record is necessary for
|
||||
# reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
|
||||
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
|
||||
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
|
||||
# each publishes so the claim is reviewable, which is all a static check can offer.
|
||||
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
|
||||
# rather than a routable address; that is an open operator decision recorded in
|
||||
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: internal-services-publish-no-public-dns
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-02: "true"
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
|
||||
# HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
|
||||
# Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
|
||||
# ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
|
||||
# at all, so a requirement-scoped selection would skip this check on precisely the changes
|
||||
# that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
A service the operator declared `internal` cannot produce a public DNS record: the
|
||||
guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
|
||||
in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
|
||||
and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
|
||||
register so the unadjudicated set cannot grow silently.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-no-internal-service-can-publish-a-public-dns-record
|
||||
description: >-
|
||||
Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
|
||||
disown annotation on every internal one and on no public one, extend the same rule to the
|
||||
`service` source, assert catalog/render.py derives the annotation from `exposure` rather
|
||||
than from a service list, prove the shipped renderer on internal/public/default-posture
|
||||
fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+458
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
|
||||
# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
|
||||
#
|
||||
# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
|
||||
# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
|
||||
# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
|
||||
# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
|
||||
# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
|
||||
# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
|
||||
# what keeps it derived, present, and scoped.
|
||||
#
|
||||
# ASSERTIONS
|
||||
# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
|
||||
# (1) every rendered route of an `internal` service carries the disown annotation;
|
||||
# (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
|
||||
# satisfy (1) and silently delete the five public records under policy: sync;
|
||||
# (3) the `service` source half: an internal service's Service may not carry an external-dns
|
||||
# hostname annotation without the disown annotation beside it;
|
||||
# (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
|
||||
# field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
|
||||
# (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
|
||||
# public, and `exposure` OMITTED — the contract's default posture is internal);
|
||||
# (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
|
||||
# KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
|
||||
# (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
|
||||
# ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
|
||||
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
|
||||
# - That the record is gone from Cloudflare. Live read; belongs beside
|
||||
# conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
|
||||
# direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
|
||||
# annotation produced no record; adding it to a route that had one produced action=DELETE for
|
||||
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
|
||||
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
|
||||
# This check owns the NAME axis only.
|
||||
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
|
||||
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
|
||||
# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
|
||||
# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
|
||||
if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
|
||||
echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
|
||||
|
||||
PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
|
||||
line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
|
||||
structures, and a line-based reader of them is precisely the approximate parser that silently
|
||||
stops matching — the failure class this check exists to prevent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
|
||||
# ControllerKey = annotationPrefix + "controller" -> the key below
|
||||
# ControllerValue = "dns-controller" -> the value that CLAIMS an object
|
||||
# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
|
||||
# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
|
||||
CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
CONTROLLER_CLAIMED = "dns-controller"
|
||||
HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
def docs_of(p: Path):
|
||||
"""Every YAML document in a file, tolerating the non-manifest files in the tree."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
|
||||
#
|
||||
# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
|
||||
# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
|
||||
# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
|
||||
# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
|
||||
# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
|
||||
# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
|
||||
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
|
||||
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
|
||||
#
|
||||
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
|
||||
# the honest state after this change, and it is deliberate: converting them is an operator
|
||||
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
|
||||
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
|
||||
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
|
||||
#
|
||||
# Keyed by "<relpath>:<namespace>/<name>".
|
||||
# =============================================================================================
|
||||
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
|
||||
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
|
||||
"platform component, not a catalog service; it must be reachable by every browser that "
|
||||
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
|
||||
"whether the five public services are — an operator decision, not a renderer one.",
|
||||
"platform/keycloak/httproute.yaml:keycloak/keycloak":
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
|
||||
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
|
||||
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
|
||||
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
|
||||
"is meant to be public either way; it is listed because the PUBLISHING object is the "
|
||||
"hand-authored platform route, which carries no exposure field.",
|
||||
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
|
||||
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
|
||||
"certainly should not be publicly resolvable, but it is not a catalog service and "
|
||||
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
|
||||
"internal services (Pi-hole must answer first).",
|
||||
"platform/headlamp/httproute.yaml:headlamp/headlamp":
|
||||
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
|
||||
"Keycloak OIDC; same adjudication as grafana.",
|
||||
"platform/homepage/httproute.yaml:homepage/homepage":
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
|
||||
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
|
||||
"declaration.",
|
||||
"runbooks/serving/httproute.yaml:runbooks/runbooks":
|
||||
"declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
|
||||
"contributes to the apex record rather than a name of its own (the apex TXT registry "
|
||||
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
|
||||
"header for why gating runbooks on Keycloak is a loop with no exit.",
|
||||
"platform/crafty/httproute.yaml:crafty/crafty":
|
||||
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
|
||||
"oauth2-proxy sidecar). Platform component, no exposure field.",
|
||||
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
|
||||
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
|
||||
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
|
||||
"Gateway cannot express. A game server is the one case where a public name may be "
|
||||
"genuinely wanted; the address it answers with is still RFC1918.",
|
||||
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
|
||||
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
|
||||
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
|
||||
"vm-demo, which declares no ingressRoutes and publishes nothing.",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
|
||||
# =============================================================================================
|
||||
CATALOG = ROOT / "catalog"
|
||||
sds: dict[str, dict] = {}
|
||||
for p in sorted(CATALOG.glob("*.yaml")):
|
||||
for d in docs_of(p):
|
||||
if "name" in d and "source" in d:
|
||||
sds[d["name"]] = d
|
||||
|
||||
check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
|
||||
|
||||
# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
|
||||
# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
|
||||
# renderer: a new service that omits exposure would go unaudited.
|
||||
exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
|
||||
# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
|
||||
# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
|
||||
expected_routes: dict[tuple[str, str], str] = {}
|
||||
for n, sd in sds.items():
|
||||
for r in sd.get("ingressRoutes") or []:
|
||||
expected_routes[(n, f"{n}-{r['port']}")] = n
|
||||
|
||||
internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "internal"})
|
||||
public_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "public"})
|
||||
check(len(internal_with_routes) > 0,
|
||||
"inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
|
||||
"this check is empty, which means either the catalog changed shape or the parse is wrong")
|
||||
check(len(public_with_routes) > 0,
|
||||
"inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
|
||||
"vacuous, so the parse is presumed wrong rather than the repo")
|
||||
|
||||
SCAN_DIRS = ["apps", "platform", "runbooks"]
|
||||
routes: list[tuple[str, dict]] = [] # (relpath, HTTPRoute doc)
|
||||
dns_services: list[tuple[str, dict]] = [] # (relpath, Service doc carrying an external-dns ann)
|
||||
for sub in SCAN_DIRS:
|
||||
base = ROOT / sub
|
||||
if not base.is_dir():
|
||||
continue
|
||||
for p in sorted(base.rglob("*.yaml")):
|
||||
rel = p.relative_to(ROOT).as_posix()
|
||||
for d in docs_of(p):
|
||||
kind = d.get("kind")
|
||||
md = d.get("metadata") or {}
|
||||
ann = md.get("annotations") or {}
|
||||
if kind == "HTTPRoute":
|
||||
routes.append((rel, d))
|
||||
elif kind == "Service" and any(
|
||||
a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
|
||||
):
|
||||
dns_services.append((rel, d))
|
||||
|
||||
check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
|
||||
"contract broken (the tree is known to author several)")
|
||||
|
||||
|
||||
def ident(d: dict) -> tuple[str, str]:
|
||||
md = d.get("metadata") or {}
|
||||
return (md.get("namespace") or "", md.get("name") or "")
|
||||
|
||||
|
||||
def annotations_of(d: dict) -> dict:
|
||||
return (d.get("metadata") or {}).get("annotations") or {}
|
||||
|
||||
|
||||
def is_disowned(d: dict) -> bool:
|
||||
"""True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
|
||||
and its value is not external-dns's own. Absent == published (the default), which is the whole
|
||||
point — the annotation is the only thing standing between a route and the public zone."""
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
return v is not None and v != CONTROLLER_CLAIMED
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# (1) + (2) every internal service's rendered route is disowned; no public one is.
|
||||
# =============================================================================================
|
||||
by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
|
||||
for rel, d in routes:
|
||||
by_ident.setdefault(ident(d), []).append((rel, d))
|
||||
|
||||
for (ns, rname), svc in sorted(expected_routes.items()):
|
||||
found = by_ident.get((ns, rname), [])
|
||||
check(len(found) >= 1,
|
||||
f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
|
||||
f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
|
||||
f"(ci/render-golden.sh owns the byte-level half of this)")
|
||||
for rel, d in found:
|
||||
if exposure_of[svc] == "internal":
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
check(
|
||||
v is not None,
|
||||
f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
|
||||
f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
|
||||
f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
|
||||
f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
|
||||
)
|
||||
check(
|
||||
v != CONTROLLER_CLAIMED,
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
|
||||
f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
|
||||
f"rather than releasing it. It reads like the guard and is its exact inverse.",
|
||||
)
|
||||
else:
|
||||
check(
|
||||
CONTROLLER_KEY not in annotations_of(d),
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
|
||||
f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
|
||||
f"`policy: sync` that DELETES its live public record — the five public services "
|
||||
f"must keep publishing.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (3) the `service` source half — an internal service's Service may not publish either.
|
||||
# =============================================================================================
|
||||
for rel, d in dns_services:
|
||||
ns, name = ident(d)
|
||||
svc = ns if ns in sds else None
|
||||
if svc and exposure_of[svc] == "internal":
|
||||
check(
|
||||
is_disowned(d),
|
||||
f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
|
||||
f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
|
||||
f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
|
||||
f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
|
||||
f"half is half a guarantee.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
|
||||
# =============================================================================================
|
||||
render_py = ROOT / "catalog" / "render.py"
|
||||
src = read(render_py)
|
||||
check(bool(src), "catalog/render.py unreadable")
|
||||
|
||||
emitter = re.search(
|
||||
r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
|
||||
)
|
||||
check(emitter is not None,
|
||||
"catalog/render.py must define `_externaldns_route_annotations` — the single place the "
|
||||
"ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
|
||||
"read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
|
||||
if emitter:
|
||||
body = emitter.group(0)
|
||||
# The derivation must read `exposure`. A body that does not mention it cannot be deriving
|
||||
# anything from the operator's declaration.
|
||||
check('"exposure"' in body or "'exposure'" in body,
|
||||
"`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
|
||||
"that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
|
||||
# and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
|
||||
# acquired one identical wrong port name in a single edit. Only the executable statements are
|
||||
# inspected — the docstring legitimately names the affected services as evidence.
|
||||
code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
|
||||
named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
|
||||
check(not named,
|
||||
f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
|
||||
f"annotation must be derived from `exposure` alone; a per-service list silently omits "
|
||||
f"the next service anyone adds.")
|
||||
check(CONTROLLER_CLAIMED not in code,
|
||||
f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
|
||||
f"external-dns's own value CLAIMS the object instead of releasing it")
|
||||
|
||||
# =============================================================================================
|
||||
# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
|
||||
# "internal is annotated" alone would also pass if the renderer annotated everything, and
|
||||
# because the contract's DEFAULT posture is internal — the case a new service hits by
|
||||
# omitting the field entirely, and the case a per-service list would miss.
|
||||
# =============================================================================================
|
||||
FIXTURES = [
|
||||
("fixture-internal", {"exposure": "internal"}, True,
|
||||
"an `internal` fixture must render the disown annotation"),
|
||||
("fixture-public", {"exposure": "public"}, False,
|
||||
"a `public` fixture must render NO disown annotation"),
|
||||
("fixture-default", {}, True,
|
||||
"a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
|
||||
"must also render the disown annotation"),
|
||||
]
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
for fname, extra, want, msg in FIXTURES:
|
||||
sd = {
|
||||
"name": fname,
|
||||
"kind": "container",
|
||||
"source": {"image": "example/app:1.0.0"},
|
||||
"egress": "internet",
|
||||
"dependencies": [],
|
||||
"ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
|
||||
# `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
|
||||
# keeping the fixture about DNS scoping and nothing else. A public service may not use
|
||||
# auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
|
||||
"auth": {"mode": "native"},
|
||||
"update": {"channel": "stable", "cadence": "0 4 * * 0",
|
||||
"autoRollbackOnFailedHealth": True},
|
||||
"observability": {"metrics": True, "slis": ["up"]},
|
||||
"backup": {"policy": "reacquirable"},
|
||||
}
|
||||
sd.update(extra)
|
||||
f = Path(td) / f"{fname}.yaml"
|
||||
f.write_text(yaml.safe_dump(sd, sort_keys=False))
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(render_py), str(f), "--no-image-automation"],
|
||||
capture_output=True, text=True, cwd=str(ROOT),
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
check(False, f"fixture {fname}: the shipped renderer failed: "
|
||||
f"{proc.stderr.strip()[:400]}")
|
||||
continue
|
||||
rendered = [d for d in yaml.safe_load_all(proc.stdout)
|
||||
if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
|
||||
check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
|
||||
if rendered:
|
||||
got = is_disowned(rendered[0])
|
||||
check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
|
||||
f"{annotations_of(rendered[0])})")
|
||||
|
||||
# =============================================================================================
|
||||
# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
|
||||
# declaration still names a real object.
|
||||
# =============================================================================================
|
||||
catalog_rendered = set(expected_routes)
|
||||
seen_register_keys: set[str] = set()
|
||||
for rel, d in routes + dns_services:
|
||||
ns, name = ident(d)
|
||||
if (ns, name) in catalog_rendered:
|
||||
continue
|
||||
key = f"{rel}:{ns}/{name}"
|
||||
seen_register_keys.add(key)
|
||||
check(
|
||||
key in KNOWN_NON_CATALOG_PUBLISHERS,
|
||||
f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
|
||||
f"`exposure` field this check can adjudicate, and it can put a name into the public "
|
||||
f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
|
||||
f"it is out of scope — or render it from a ServiceDefinition instead.",
|
||||
)
|
||||
stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
|
||||
check(not stale,
|
||||
f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
|
||||
f"name no object in the tree any more. Delete them — a register that keeps entries for "
|
||||
f"deleted objects stops being evidence of anything.")
|
||||
|
||||
# =============================================================================================
|
||||
# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
|
||||
# always selects this test) and NO stage is hand-written into ci/l0.sh (the
|
||||
# whole-repo-invariants-wired meta-test fails the gate if one is).
|
||||
# =============================================================================================
|
||||
TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
|
||||
test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
|
||||
check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
|
||||
check(
|
||||
re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
|
||||
is not None,
|
||||
"chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
|
||||
"mapping line — that declaration is the only thing that unions this check into every gate "
|
||||
"run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
|
||||
)
|
||||
l0_sh = read(ROOT / "ci/l0.sh")
|
||||
check(
|
||||
"internal-services-publish-no-public-dns" not in l0_sh,
|
||||
"ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
|
||||
"and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
print(f"check FAIL: {f}", file=sys.stderr)
|
||||
print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: {CHECKS} assertions passed "
|
||||
f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
|
||||
f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
|
||||
f"{len(dns_services)} external-dns-annotated Services inventoried; "
|
||||
f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
|
||||
PY
|
||||
@@ -238,7 +238,7 @@ def bail(msg):
|
||||
|
||||
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
|
||||
SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
|
||||
SKIP_DIRS = {".git", "_artifacts", "node_modules", ".venv"}
|
||||
SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
|
||||
|
||||
# ------------------------------------------------------------------ discover the guard FILE
|
||||
# Hardcoding a path is what let a file that never landed disable this whole check.
|
||||
|
||||
@@ -154,8 +154,13 @@ services: list[tuple] = []
|
||||
monitors: list[tuple] = []
|
||||
unparseable: list[str] = []
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts or is_test_fixture(p) or ".sops." in p.name:
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
|
||||
continue
|
||||
txt = read(p)
|
||||
if not txt:
|
||||
|
||||
@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
|
||||
fi
|
||||
|
||||
grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH"
|
||||
chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
"conformance/lib/common.sh no longer resolves the scope annotation — no test can be
|
||||
selected BY declaring a scope, so every whole-repo invariant is back to being
|
||||
selectable away by the requirement filter"
|
||||
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
|
||||
chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
"conformance/run.sh does not union scope-declared tests (--include-scope /
|
||||
conf_dirs_for_scopes missing) — the annotation would be inert decoration"
|
||||
|
||||
grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
|
||||
chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
"ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
|
||||
purely requirement-scoped run, and every whole-repo invariant is filtered out of
|
||||
exactly the change that breaks it"
|
||||
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
|
||||
CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
|
||||
)
|
||||
MIN_DECLARED="${MIN_DECLARED:-5}"
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
|
||||
chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
|
||||
chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
"only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
|
||||
annotation was stripped from the whole-repo checks, or the resolver stopped matching
|
||||
it. Every assertion below would then pass while checking nothing"
|
||||
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
rel="${d#"$ROOT"/}"
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
|
||||
chk $? "${rel}: declares the scope AND carries an executable offline check" \
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
|
||||
chk $arc "${rel}: declares the scope AND carries an executable offline check" \
|
||||
"${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
|
||||
unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
|
||||
permanent not-green, not a guard"
|
||||
@@ -91,25 +91,58 @@ BOGUS_REQ="r-nosuchreq-99"
|
||||
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
|
||||
sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
|
||||
|
||||
[ -z "$sel_without" ]
|
||||
chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
[ -z "$sel_without" ] && arc=0 || arc=$?
|
||||
chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
"the control case is broken: --requirements ${BOGUS_REQ} still selected tests
|
||||
($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
|
||||
about --include-scope"
|
||||
|
||||
# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
|
||||
# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
|
||||
# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
|
||||
# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
|
||||
# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
|
||||
# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
|
||||
missing=""
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
|
||||
printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ -z "$missing" ]
|
||||
chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
[ -z "$missing" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
"the scope union does not actually select:${missing}. This is the blind spot itself: a
|
||||
change that attributes no requirement would not run these checks"
|
||||
|
||||
# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
|
||||
# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
|
||||
# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
|
||||
# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
|
||||
# three rendering goldens stale.
|
||||
sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
|
||||
--print-selection 2>/dev/null || true)"
|
||||
missing_l1=""
|
||||
declared_l1=0
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
|
||||
declared_l1=$((declared_l1+1))
|
||||
printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
|
||||
"scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
|
||||
${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
|
||||
when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
|
||||
|
||||
# And the L1 union must not leak the other way either.
|
||||
strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
|
||||
[ -z "$strays_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
|
||||
"--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
|
||||
|
||||
# The union must not smuggle in another layer's tests when --layer bounds the run.
|
||||
strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
|
||||
[ -z "$strays" ]
|
||||
chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
[ -z "$strays" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
"--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
|
||||
|
||||
# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
|
||||
@@ -143,23 +176,23 @@ run_offline() { # run_offline <dir...> -> echoes rc
|
||||
rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
|
||||
rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
|
||||
|
||||
[ "$rc_declared" = 3 ]
|
||||
chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
[ "$rc_declared" = 3 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
"a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
|
||||
whole-repo invariant reporting 'nothing to audit' means the tree it audits has
|
||||
VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
|
||||
rc=3. Exit 0 here turns that back into a silent green"
|
||||
|
||||
[ "$rc_undeclared" = 0 ]
|
||||
chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
"a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
|
||||
rule leaked onto ordinary scaffolds, which would make every unfinished test block"
|
||||
|
||||
# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
|
||||
l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
|
||||
printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
|
||||
chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
"ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
|
||||
check that SKIPPED would be reported as a PASS — the exact false green the deleted
|
||||
stages were written to prevent"
|
||||
@@ -193,8 +226,8 @@ done
|
||||
|
||||
# ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
|
||||
# Carried over from the three deleted stages' own checks — asserted once, here.
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
|
||||
chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
"ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
|
||||
would hold while a red L0 stage merged anyway"
|
||||
|
||||
|
||||
@@ -23,6 +23,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-PLAT-02"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
|
||||
spec:
|
||||
|
||||
@@ -196,6 +196,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -27,6 +27,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
Rendering any service's namespace yields a default-deny baseline policy.
|
||||
spec:
|
||||
|
||||
@@ -32,6 +32,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
egress: vpn renders the egress-gateway + kill-switch path; internet does not.
|
||||
spec:
|
||||
|
||||
+1
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
|
||||
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
|
||||
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
|
||||
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
|
||||
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
|
||||
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
|
||||
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
|
||||
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
|
||||
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
|
||||
> 2026-09-29, lie about it.
|
||||
>
|
||||
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
|
||||
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
|
||||
> not enable L2/L3, it only changes which way this stage fails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
# infra/gitea — platform settings that are NOT GitOps-managed
|
||||
|
||||
Gitea's own repo settings (branch protection, required status checks, approvals) live
|
||||
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
|
||||
and invisible to the gate** — exactly how the state below went unnoticed.
|
||||
|
||||
## The gap this closes
|
||||
|
||||
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
|
||||
(2026-08-31) asserting the opposite:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
|
||||
|
||||
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
|
||||
rule; this directory is how the rule gets enforced by the machine.
|
||||
|
||||
## `branch-protection.sh`
|
||||
|
||||
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
|
||||
do not trust the API response**, since Gitea may let site admins bypass protection.
|
||||
|
||||
./infra/gitea/branch-protection.sh
|
||||
|
||||
It authenticates with the credential `git` already has stored for this host, and never
|
||||
prints the secret.
|
||||
|
||||
## `runner-token-rotate.sh`
|
||||
|
||||
Mints a fresh instance-level registration token from the Gitea API and writes it into
|
||||
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
|
||||
never printed.**
|
||||
|
||||
./infra/gitea/runner-token-rotate.sh
|
||||
|
||||
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
|
||||
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
|
||||
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
|
||||
and polls with an identity Gitea does not recognise — so it never appears in the runner
|
||||
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
|
||||
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
||||
commands.
|
||||
|
||||
## Sequence — DONE 2026-09-29
|
||||
|
||||
All four steps are complete and each was verified, not assumed:
|
||||
|
||||
| Step | State |
|
||||
|---|---|
|
||||
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
||||
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
||||
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
||||
|
||||
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
||||
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
||||
account agents authenticate as.
|
||||
|
||||
### 5. Block the admin override — the step the first pass MISSED
|
||||
|
||||
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
|
||||
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
|
||||
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
|
||||
like the gate were overridable by precisely the account that merges.
|
||||
|
||||
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
|
||||
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
|
||||
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
|
||||
(`/branch_protections/main`) and enumerate its keys.
|
||||
|
||||
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
|
||||
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
|
||||
> means "key absent". Enumerate with `to_entries` and act on what is really there.
|
||||
|
||||
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
|
||||
proven when a green PR is REFUSED a merge until approved, and not before.
|
||||
|
||||
### `require-gate.sh` / `verify-then-enforce.sh`
|
||||
|
||||
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
|
||||
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
|
||||
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
|
||||
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
|
||||
|
||||
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
||||
|
||||
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
||||
|
||||
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
||||
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
||||
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
||||
database.
|
||||
|
||||
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
||||
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
||||
|
||||
Behaviour, proven by test rather than assumed:
|
||||
|
||||
| Situation | Exit | Effect |
|
||||
|---|---|---|
|
||||
| live matches the declaration | 0 | green |
|
||||
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
||||
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
||||
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
||||
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
||||
`gate.yml` so a future missing token blocks instead of skipping.
|
||||
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
conformance gate can read. Nothing detects drift from what this directory declares. That is
|
||||
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
|
||||
gone for a month while the log recorded both as working, and **no check existed that could
|
||||
have caught it.**
|
||||
|
||||
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
||||
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
||||
root cause of the 2026-09-29 session.
|
||||
|
||||
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. That is the remaining step.
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
### Original ordering notes
|
||||
|
||||
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
||||
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
||||
skip the confirmation: the whole reason this went unnoticed is that a runner was
|
||||
*recorded* as online without anyone checking.
|
||||
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
||||
3. **Only then set `enable_status_check: true`** with that exact context.
|
||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||
|
||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
||||
> reported.
|
||||
|
||||
### Measured, once the runner came up (2026-09-29)
|
||||
|
||||
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
||||
|
||||
| Context | Result | Time |
|
||||
|---|---|---|
|
||||
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
||||
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
||||
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
||||
|
||||
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
||||
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
||||
does not hang a status check: Gitea reports it as **`success`** with the description
|
||||
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
||||
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
||||
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
||||
exactly the fail-open shape this repo keeps finding.
|
||||
|
||||
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
||||
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
||||
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
||||
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stop repository admins from overriding the merge requirements on `main`.
|
||||
#
|
||||
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
|
||||
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
|
||||
# required_approvals and the required status check were both overridable by the one account
|
||||
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
|
||||
# `required_approvals: 1`.
|
||||
#
|
||||
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
|
||||
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
|
||||
# real keys (`to_entries`) rather than probing names you assume exist.
|
||||
#
|
||||
# Recoverable: an owner can PATCH this back to false if it gets in the way.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
|
||||
|
||||
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
|
||||
api -X PATCH -H 'Content-Type: application/json' \
|
||||
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
|
||||
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
|
||||
echo "==> after: block_admin_merge_override = $AFTER"
|
||||
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
|
||||
|
||||
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
|
||||
cat <<'NOTE'
|
||||
|
||||
NOW PROVE IT -- the setting is not the evidence. On the next PR:
|
||||
* let the gate go green, then try to merge WITHOUT approving it.
|
||||
* Gitea must REFUSE. If it merges, this still is not doing its job.
|
||||
Do not record approvals as enforced until you have seen a merge refused.
|
||||
NOTE
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# Enable branch protection on main for home-network-k3s-impl.
|
||||
#
|
||||
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
|
||||
#
|
||||
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
|
||||
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
|
||||
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
|
||||
# Turn it on only after a runner is live and you have seen the gate actually report on
|
||||
# a real PR, using the context name CI itself produced.
|
||||
#
|
||||
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
|
||||
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
|
||||
# Add the collaborator first, then raise this to 1.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
|
||||
echo "==> creating branch protection on 'main'"
|
||||
jq -n '{
|
||||
branch_name: "main",
|
||||
enable_push: false,
|
||||
enable_status_check: false,
|
||||
status_check_contexts: [],
|
||||
required_approvals: 0,
|
||||
block_on_rejected_reviews: true,
|
||||
block_on_outdated_branch: false,
|
||||
dismiss_stale_approvals: true
|
||||
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
-X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
|
||||
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
|
||||
|
||||
echo "==> verifying (this is the part that matters)"
|
||||
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Now prove it actually blocks you -- do not take the API's word for it. From a clean
|
||||
checkout on main, with nothing to lose:
|
||||
|
||||
git commit --allow-empty -m 'protection probe'
|
||||
git push origin main # EXPECT: rejected by the remote
|
||||
git reset --hard HEAD~1 # tidy up either way
|
||||
|
||||
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
|
||||
yet doing what you want.
|
||||
NOTE
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
"enable_push": false,
|
||||
"enable_push_whitelist": false,
|
||||
"enable_merge_whitelist": false,
|
||||
"enable_status_check": true,
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
"dismiss_stale_approvals": true
|
||||
}
|
||||
}
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
|
||||
#
|
||||
# RUN THIS LAST. Prerequisites, in order:
|
||||
# a) PR #7 (dind native sidecar) is MERGED, and
|
||||
# b) you have SEEN a real workflow run go green on the converted pod.
|
||||
# The runner Deployment is the merge gate for the whole repo. Turning on a required check
|
||||
# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
|
||||
# an owner can set enable_status_check back to false. But do not rely on that.)
|
||||
#
|
||||
# Requires exactly ONE context: `gate / gate (pull_request)`.
|
||||
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
|
||||
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
|
||||
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
|
||||
# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
|
||||
# lets repo admins override merge requirements without it, so required_approvals and the
|
||||
# required status check are both bypassable by exactly the account that does the merging.
|
||||
# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
|
||||
#
|
||||
# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
|
||||
# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
|
||||
# instead of probing names you assume exist. That mistake cost a wasted PATCH.
|
||||
#
|
||||
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
|
||||
# PR would never produce that context.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> updating branch protection on main"
|
||||
jq -n '{
|
||||
enable_push: false,
|
||||
enable_status_check: true,
|
||||
status_check_contexts: ["gate / gate (pull_request)"],
|
||||
required_approvals: 1,
|
||||
block_on_rejected_reviews: true,
|
||||
dismiss_stale_approvals: true,
|
||||
block_on_outdated_branch: false,
|
||||
block_admin_merge_override: true
|
||||
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
|
||||
|
||||
echo "==> verifying"
|
||||
api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
|
||||
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
|
||||
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
|
||||
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
|
||||
and rely on the status check alone.
|
||||
|
||||
Then prove it, rather than trusting the API:
|
||||
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
|
||||
git reset --hard HEAD~1
|
||||
NOTE
|
||||
Executable
+61
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env bash
|
||||
# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
|
||||
#
|
||||
# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
|
||||
# nowhere else. (It is briefly visible in this host's process list as a sops argument;
|
||||
# acceptable on a single-operator workstation, and far better than echoing it.)
|
||||
#
|
||||
# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
|
||||
echo "==> minting an instance-level registration token as '${U}'"
|
||||
|
||||
TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
echo "==> writing it into ${SECRET} via sops"
|
||||
DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
|
||||
sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
|
||||
unset TOKEN
|
||||
|
||||
echo "==> done. Verify the file is still encrypted (no plaintext token):"
|
||||
grep -q 'ENC\[' "$SECRET" && echo " OK - stringData is still SOPS-encrypted" \
|
||||
|| { echo " REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
|
||||
|
||||
IMPORTANT -- a new token is usually NOT sufficient on its own.
|
||||
act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
|
||||
act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
|
||||
deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
|
||||
polls with an identity Gitea does not recognise -- so it never appears in the runner list,
|
||||
which is exactly the observed symptom (zero runners, pod running).
|
||||
|
||||
So after the Secret lands, force a genuine re-registration by clearing that state:
|
||||
|
||||
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
|
||||
kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
|
||||
kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
|
||||
|
||||
(The PVC holds only runner registration state -- no build artifacts, nothing durable.
|
||||
Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
|
||||
|
||||
Then confirm it actually registered -- do not assume:
|
||||
|
||||
curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
|
||||
|
||||
A runner must appear with status "online" and the labels ubuntu-latest / homelab.
|
||||
NOTE
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
|
||||
# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
|
||||
#
|
||||
# The token value is NEVER printed. It goes from the token API into the secrets API and
|
||||
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
|
||||
#
|
||||
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
|
||||
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
|
||||
# confirm the check goes green rather than assuming the scope name.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
|
||||
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
|
||||
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
|
||||
SECRET=SETTINGS_API_TOKEN
|
||||
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> minting token '${NAME}' as ${U}"
|
||||
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
|
||||
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
|
||||
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
echo "==> storing it as repo Actions secret ${SECRET}"
|
||||
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
|
||||
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
|
||||
unset TOKEN
|
||||
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
|
||||
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
|
||||
rm -f /tmp/set-ci-token.out; exit 1
|
||||
fi
|
||||
rm -f /tmp/set-ci-token.out
|
||||
|
||||
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
|
||||
# reports success for a store that never happened -- which is exactly what it did.
|
||||
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
|
||||
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
|
||||
grep -q "$SECRET" <<<"$names" \
|
||||
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
|
||||
echo "==> stored and VERIFIED present: ${names}"
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Next:
|
||||
1. .gitea/workflows/gate.yml already passes this secret to the gate step.
|
||||
2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
|
||||
without the token, so a green run is the only proof the wiring works.
|
||||
3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
|
||||
instead of skipping. Do that only after you have seen it green, or you lock the repo.
|
||||
NOTE
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
|
||||
#
|
||||
# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
|
||||
# merges). This script does the mechanical half: prove the converted runner actually works,
|
||||
# and only then turn on enforcement.
|
||||
#
|
||||
# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
|
||||
# required status check while the runner is broken means NO PR can merge -- including the
|
||||
# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
|
||||
# only then does enforcement go on.
|
||||
#
|
||||
# Aborts at the first checkpoint that does not pass. Safe to re-run.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
|
||||
D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
|
||||
die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
|
||||
|
||||
say "PRECHECK — all four PRs must be merged first"
|
||||
for n in 8 5 6 7; do
|
||||
m=$(api "$A/pulls/$n" | jq -r .merged)
|
||||
printf ' PR #%s merged=%s\n' "$n" "$m"
|
||||
[ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
|
||||
done
|
||||
|
||||
say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
|
||||
|
||||
say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
|
||||
./infra/devcontainer/dev.sh run bash -c '
|
||||
kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
|
||||
echo "--- sidecar (initContainers) ---"
|
||||
kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
|
||||
-o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
|
||||
kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
|
||||
|| die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
|
||||
|
||||
say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
|
||||
n=0
|
||||
for i in $(seq 1 40); do
|
||||
n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
|
||||
printf '\r online runners: %s (%ss) ' "$n" "$((i*5))"
|
||||
[ "${n:-0}" -ge 1 ] && { echo; break; }
|
||||
sleep 5
|
||||
done
|
||||
[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
|
||||
|
||||
say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
|
||||
MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
|
||||
echo " watching the post-merge push run on main (${MAIN_SHA:0:8})..."
|
||||
st=""
|
||||
for i in $(seq 1 60); do
|
||||
st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
|
||||
printf '\r main gate: %-8s (%ss) ' "$st" "$((i*10))"
|
||||
case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
|
||||
sleep 10
|
||||
done
|
||||
[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
|
||||
echo " the converted sidecar ran a real job green. Enforcement is now safe."
|
||||
|
||||
say "STEP 5 — require the gate + one approval"
|
||||
"$D/require-gate.sh"
|
||||
|
||||
say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
|
||||
git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
|
||||
git commit -q --allow-empty -m 'protection probe'
|
||||
if git push origin main 2>&1 | grep -q 'protected branch'; then
|
||||
echo " CONFIRMED: push to main rejected by the pre-receive hook."
|
||||
else
|
||||
echo " WARNING: the push was NOT rejected -- protection is not doing what it claims."
|
||||
fi
|
||||
git reset -q --hard origin/main
|
||||
|
||||
say "FINAL STATE"
|
||||
api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env bash
|
||||
# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
|
||||
# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
|
||||
# stopwatch so you can see the budget being spent.
|
||||
#
|
||||
# branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
|
||||
#
|
||||
# It never sees, prints, or handles the token: the file is already encrypted by sops.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
SECRET=apps/wave-0/plex-claim-secret.sops.yaml
|
||||
T0=$(date +%s)
|
||||
el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
# --- preflight: refuse to ship a file that is not what we think it is -------------
|
||||
git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
|
||||
echo " infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
|
||||
grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
|
||||
[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
|
||||
# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
|
||||
# matching prefix fired a redundant run that serialised ahead of the PR run. Those
|
||||
# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
|
||||
BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
|
||||
|
||||
el "committing on $BRANCH"
|
||||
git checkout -q -b "$BRANCH"
|
||||
git add "$SECRET"
|
||||
git commit -q -m 'chore(plex): fresh plex.tv claim token
|
||||
|
||||
Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
|
||||
SHA=$(git rev-parse HEAD)
|
||||
|
||||
el "pushing"
|
||||
git push -q -u origin "$BRANCH"
|
||||
|
||||
el "opening PR"
|
||||
PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
|
||||
'{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
|
||||
| api -X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
|
||||
[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
|
||||
echo " PR #$PR"
|
||||
|
||||
el "waiting for the gate (expect ~1m40s)"
|
||||
for i in $(seq 1 60); do
|
||||
st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
|
||||
printf '\r gate=%s [T+%ss] ' "$st" "$(( $(date +%s)-T0 ))"
|
||||
case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
|
||||
sleep 5
|
||||
done
|
||||
[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
|
||||
|
||||
el "merging PR #$PR"
|
||||
api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
|
||||
git checkout -q main && git pull -q --ff-only origin main
|
||||
|
||||
el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
|
||||
|
||||
# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
|
||||
# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
|
||||
# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
|
||||
el "rolling the pod (mandatory — a changed Secret restarts nothing)"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
|
||||
kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
|
||||
|
||||
el "verdict"
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
|
||||
cat <<'NOTE'
|
||||
|
||||
"Attempting to obtain server token..." + "Token obtained successfully" -> CLAIMED
|
||||
"Attempting..." with no success line -> token expired; retry, nothing is broken
|
||||
neither line -> already claimed, or env did not reach the container
|
||||
|
||||
Confirm it persisted (prints the attribute NAME only, never its value):
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
|
||||
"grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
|
||||
NOTE
|
||||
@@ -34,6 +34,14 @@ resources:
|
||||
# policies, the next import is blocked in the gap.
|
||||
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
|
||||
- ../platform/kyverno/admission/secret-provenance.yaml
|
||||
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
|
||||
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
|
||||
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
|
||||
# only ("kept as reference; THIS file is the buildable render root" in
|
||||
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
|
||||
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
|
||||
# leaving the conformance-trigger identity live with nothing bounding it.
|
||||
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
|
||||
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
|
||||
- cilium/l2-ip-pool.yaml
|
||||
- cilium/l2-announcement-policy.yaml
|
||||
|
||||
@@ -26,10 +26,94 @@ spec:
|
||||
provider:
|
||||
name: cloudflare
|
||||
# Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
|
||||
#
|
||||
# BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block
|
||||
# below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list.
|
||||
sources:
|
||||
- gateway-httproute
|
||||
- service
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14)
|
||||
#
|
||||
# THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's
|
||||
# address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
||||
# public internet and that Pi-hole serves the internal split-horizon answers; the renderer
|
||||
# contract said an `internal` service renders "no public DNS record". Neither sentence was
|
||||
# enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent,
|
||||
# radarr, semaphore, sonarr) published `<name>.home.keithjackson.dev` into this PUBLIC
|
||||
# authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP
|
||||
# and a complete service inventory to anyone who could guess a label, and made every one of
|
||||
# those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1
|
||||
# strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own
|
||||
# services. Pre-state captured at the AUTHORITY and committed:
|
||||
# .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
||||
#
|
||||
# THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`.
|
||||
# An `internal` service's rendered HTTPRoute carries
|
||||
# external-dns.alpha.kubernetes.io/controller: none
|
||||
# ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image
|
||||
# registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller`
|
||||
# annotation is present and != its own `dns-controller`:
|
||||
# source/annotations/processors.go IsControllerMismatch()
|
||||
# source/gateway.go Endpoints(): if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
||||
# source/service.go Endpoints(): the identical check
|
||||
# The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an
|
||||
# endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway
|
||||
# hostname and no existing record at risk: an annotated route published nothing at all, and
|
||||
# ADDING the annotation to a route that already had a record produced
|
||||
# action=DELETE record=<name> type=A and action=DELETE record=a-<name> type=TXT
|
||||
# one ~60s reconcile later, and moved nothing else in the zone.
|
||||
#
|
||||
# WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter`
|
||||
# (--label-filter) is the stronger default posture — publish only what is labelled — and
|
||||
# v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by
|
||||
# source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here
|
||||
# because its blast radius is not this defect's. Nine publishing objects in this repo are
|
||||
# hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and
|
||||
# seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp,
|
||||
# the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile,
|
||||
# and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a
|
||||
# renderer one. They are enumerated in the shrink-only register in
|
||||
# conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set
|
||||
# cannot grow silently — adjudicate them there, then this filter becomes the right hardening.
|
||||
#
|
||||
# DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
|
||||
#
|
||||
# The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
|
||||
# that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
|
||||
# actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
|
||||
# this change, for two independent reasons:
|
||||
#
|
||||
# 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
|
||||
# (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
|
||||
# points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
|
||||
# certificate. Whether prod's public services belong behind it, behind a new forward, or
|
||||
# are not meant to be public at all is an operator decision. Breaking five services' DNS
|
||||
# on a guess is worse than leaving a known-wrong-but-stable state.
|
||||
# 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
|
||||
# real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
|
||||
# GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
|
||||
# no per-service target override while every public service shares the `shared` Gateway.
|
||||
# Annotating the Gateway would move EVERY gateway-sourced record, which after this change
|
||||
# still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
|
||||
# routable address for admin UIs. That is strictly worse than the status quo.
|
||||
#
|
||||
# SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
|
||||
# is answered, because a knob that cannot take effect invites someone to set it and expect one:
|
||||
# (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
|
||||
# port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
|
||||
# (b) HOW should they be separated from the non-public names that share the Gateway? Options,
|
||||
# roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
|
||||
# annotation, with only the public services' routes attached; or publishing the five names
|
||||
# from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
|
||||
# which restores per-service targeting without a second Gateway.
|
||||
# Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
# On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
# Serves: R-SEC-02 (CI holds no broad cluster access)
|
||||
#
|
||||
# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
|
||||
#
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
|
||||
# `create jobs` in namespace `conformance` can therefore create a Job that runs as
|
||||
# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
|
||||
# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
|
||||
# privileged identity unless something constrains the SHAPE of what it may create.
|
||||
#
|
||||
# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
|
||||
# committed conformance run and deny everything else. The operator chose this model
|
||||
# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
|
||||
# GitOps-native alternative.
|
||||
#
|
||||
# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
|
||||
# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
|
||||
# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
|
||||
# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
|
||||
# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
|
||||
#
|
||||
# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
|
||||
# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
|
||||
#
|
||||
# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
|
||||
# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
|
||||
# enforceable half: the image must live under the conformance repository path AND be
|
||||
# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
|
||||
# registry path could publish a different digest. TIGHTEN THIS when the image is built by
|
||||
# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
|
||||
# PR, which is exactly where a change to the security boundary belongs.
|
||||
apiVersion: kyverno.io/v1
|
||||
kind: ClusterPolicy
|
||||
metadata:
|
||||
name: conformance-trigger-job-shape
|
||||
annotations:
|
||||
policies.kyverno.io/title: Constrain what the conformance CI trigger may create
|
||||
policies.kyverno.io/category: Supply chain
|
||||
policies.kyverno.io/severity: critical
|
||||
homelab.keithjackson.dev/requirements: R-SEC-02
|
||||
homelab.keithjackson.dev/scope: admission-only
|
||||
spec:
|
||||
validationFailureAction: Enforce
|
||||
failurePolicy: Fail
|
||||
background: false
|
||||
rules:
|
||||
- name: trigger-may-create-only-the-committed-conformance-run
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger may create ONLY the committed conformance run: a single container,
|
||||
digest-pinned conformance-runner image, serviceAccountName conformance-runner,
|
||||
command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
|
||||
and only the `workspace` and `results` volumes. See
|
||||
platform/kyverno/admission/conformance-trigger-job-shape.yaml.
|
||||
pattern:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
=(hostNetwork): "false"
|
||||
=(hostPID): "false"
|
||||
=(hostIPC): "false"
|
||||
containers:
|
||||
- image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
|
||||
command:
|
||||
- "./run.sh"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
=(privileged): "false"
|
||||
|
||||
- name: trigger-volumes-must-be-workspace-or-results
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
|
||||
volume is how a constrained pod reaches a host path, a secret, or another SA's token.
|
||||
# foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
|
||||
# matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
|
||||
# asserts EVERY volume is one of the two allowed names.
|
||||
foreach:
|
||||
- list: "request.object.spec.template.spec.volumes"
|
||||
deny:
|
||||
conditions:
|
||||
all:
|
||||
- key: "{{ element.name }}"
|
||||
operator: AnyNotIn
|
||||
value: ["workspace", "results"]
|
||||
|
||||
- name: trigger-args-must-be-the-committed-flags
|
||||
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
|
||||
# As two separate `all` entries it is a schema error ("match resources not specified")
|
||||
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- conformance
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
validate:
|
||||
message: >-
|
||||
conformance-trigger Job args may contain only the conformance run's own flags
|
||||
(--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
|
||||
Anything else — a shell, a path, a redirect — is denied.
|
||||
foreach:
|
||||
- list: "request.object.spec.template.spec.containers[0].args"
|
||||
deny:
|
||||
conditions:
|
||||
all:
|
||||
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
|
||||
operator: Equals
|
||||
value: false
|
||||
@@ -20,3 +20,7 @@ resources:
|
||||
# --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
|
||||
# (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
|
||||
- secret-provenance.yaml
|
||||
# --- R-SEC-02: constrain what the conformance CI trigger may create ---
|
||||
# THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
|
||||
# subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
|
||||
- conformance-trigger-job-shape.yaml
|
||||
@@ -0,0 +1,57 @@
|
||||
# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
|
||||
#
|
||||
# Each DENY case below is a real escalation path, not a style violation. The one that matters
|
||||
# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
|
||||
# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
|
||||
# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
|
||||
# this policy.
|
||||
#
|
||||
# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
|
||||
# never fires and every case trivially passes — a test that proves nothing.
|
||||
apiVersion: cli.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: conformance-trigger-job-shape
|
||||
policies:
|
||||
- ../../admission/conformance-trigger-job-shape.yaml
|
||||
resources:
|
||||
- resources.yaml
|
||||
userinfo: user-info.yaml
|
||||
results:
|
||||
# ---- the committed run is ADMITTED ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-may-create-only-the-committed-conformance-run
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-volumes-must-be-workspace-or-results
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-args-must-be-the-committed-flags
|
||||
kind: Job
|
||||
resources: [good-run]
|
||||
result: pass
|
||||
|
||||
# ---- THE escalation: runner SA + arbitrary command ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-may-create-only-the-committed-conformance-run
|
||||
kind: Job
|
||||
resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
|
||||
result: fail
|
||||
|
||||
# ---- an extra volume is the way out of a constrained pod ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-volumes-must-be-workspace-or-results
|
||||
kind: Job
|
||||
resources: [extra-volume]
|
||||
result: fail
|
||||
|
||||
# ---- args beyond the committed flags ----
|
||||
- policy: conformance-trigger-job-shape
|
||||
rule: trigger-args-must-be-the-committed-flags
|
||||
kind: Job
|
||||
resources: [injected-args]
|
||||
result: fail
|
||||
@@ -0,0 +1,154 @@
|
||||
# The committed shape — must be ADMITTED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: good-run
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
- name: results
|
||||
emptyDir: {}
|
||||
---
|
||||
# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evil-command
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# An arbitrary image running as the privileged SA. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evil-image
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: docker.io/library/alpine:latest
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: tag-not-digest
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# Host namespaces + privilege escalation. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: host-namespaces
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
hostPID: true
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: false
|
||||
allowPrivilegeEscalation: true
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
---
|
||||
# An extra volume — the way out of a constrained pod. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: extra-volume
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
- name: hostroot
|
||||
hostPath:
|
||||
path: /
|
||||
---
|
||||
# Injected args beyond the committed flags. Must be DENIED.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: injected-args
|
||||
namespace: conformance
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: conformance-runner
|
||||
containers:
|
||||
- name: runner
|
||||
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["./run.sh"]
|
||||
args: ["--layer", "l2", ";", "curl", "http://evil/"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
@@ -0,0 +1,12 @@
|
||||
# The requesting identity the policy keys on. Without this the `subjects` match never fires and
|
||||
# every case below would trivially "pass" — proving nothing.
|
||||
apiVersion: cli.kyverno.io/v1alpha1
|
||||
kind: UserInfo
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
userInfo:
|
||||
username: system:serviceaccount:conformance:conformance-trigger
|
||||
groups:
|
||||
- system:serviceaccounts
|
||||
- system:serviceaccounts:conformance
|
||||
- system:authenticated
|
||||
@@ -904,6 +904,132 @@ spec:
|
||||
summary: "Egress VPN tunnel down"
|
||||
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
|
||||
#
|
||||
# `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
|
||||
# having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
|
||||
# exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
|
||||
# re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
|
||||
# at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
|
||||
# 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
|
||||
# carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
|
||||
# exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
|
||||
#
|
||||
# The datapath now self-heals: `roles/vpn_egress_gateway` publishes
|
||||
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
|
||||
# OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
|
||||
# and that node write is what makes Cilium re-resolve. These two rules watch whether it
|
||||
# ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
|
||||
#
|
||||
# WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
|
||||
# `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
|
||||
# observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
|
||||
# * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
|
||||
# `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
|
||||
# there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
|
||||
# metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
|
||||
# `cilium_*` and `hubble_*` query returns an empty vector today (which also means
|
||||
# `HubbleDeniedFlowSpike` above cannot fire yet).
|
||||
# * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
|
||||
# CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
|
||||
# map, not an exporter. No metric name is invented here to paper over that.
|
||||
# So the observable is the label: the input to the re-steer, compared against the live tun0
|
||||
# identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
|
||||
# ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
|
||||
# `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
|
||||
#
|
||||
# WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
|
||||
# firing, the label write failing or never being deployed, the convergence timer being dead —
|
||||
# i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
|
||||
# "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
|
||||
# until either the re-steer script exports the map's ifindex through a node-exporter textfile
|
||||
# collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
|
||||
# A probe was considered and rejected for now: there is no blackbox-exporter and no probe
|
||||
# CronJob in this repo, so it means a NEW always-on component whose own failures become a new
|
||||
# alerting surface — worth doing, but not as the thing that closes a hole today.
|
||||
#
|
||||
# WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
|
||||
# VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
|
||||
# label-to-number conversion, so the level-triggered equality test is not expressible. What IS
|
||||
# expressible is the same invariant counted over a window: the published label must change at
|
||||
# least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
|
||||
# is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
|
||||
# that held still, 2 for one that changed value inside the window); the `max by (<label>)`
|
||||
# wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
|
||||
# restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
|
||||
# which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
|
||||
#
|
||||
# Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
|
||||
# +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
|
||||
# trade against the repo's alert-fatigue history (an unbounded critical on the one channel
|
||||
# that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
|
||||
# email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
|
||||
- alert: VPNEgressSteeringStale
|
||||
# FIRES when the live tun0 was re-created more times in the last hour than the gateway
|
||||
# node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
|
||||
# tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
|
||||
# and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
|
||||
# seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
|
||||
# cold boot), so `for: 15m` cannot fire on a healthy reconnect.
|
||||
expr: |
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(
|
||||
count(
|
||||
count_over_time(
|
||||
max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
|
||||
)[1h:1m]
|
||||
)
|
||||
) - 1
|
||||
)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare
|
||||
`kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
|
||||
`cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
|
||||
(or setting any node label) forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
- alert: VPNEgressUplinkLabelMissing
|
||||
# The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
|
||||
# in kube-state-metrics, and if that label is missing its comparison degrades to an empty
|
||||
# vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
|
||||
# node with no published uplink means the re-steer mechanism is not in place at all
|
||||
# (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
|
||||
# kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
|
||||
# the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
|
||||
# rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
|
||||
# that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
|
||||
# on the way in, and so an Ansible run in progress does not either.
|
||||
expr: |
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
and on()
|
||||
absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
|
||||
nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
|
||||
`VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -231,7 +231,7 @@ spec:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
@@ -116,10 +116,14 @@ fi
|
||||
# `.git` file points outside the mount, so git cannot read the index there — and with
|
||||
# `pipefail` that failure silently emptied the discovery, i.e. a green run over ZERO tests
|
||||
# (observed 2026-09-12). `find` needs no repo metadata, so discovery cannot be disabled by
|
||||
# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up.
|
||||
# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up,
|
||||
# and .claude/ with it: .claude/worktrees/agent-*/ holds full repo copies for isolated agent
|
||||
# runs, so an unpruned find reported the SAME manifest under test once per worktree (up to 18
|
||||
# times), and the L0 check that compares this listing against the tree failed on paths that
|
||||
# are not authored surface at all (.claude/ is gitignored, .gitignore:43).
|
||||
discover() {
|
||||
# Companion convention, anywhere in the tree.
|
||||
find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
|
||||
find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
|
||||
# Suite convention, ANYWHERE in the tree — not just this directory.
|
||||
#
|
||||
# This used to be `find "${HERE}" -maxdepth 1`, which made the suite convention a
|
||||
@@ -129,7 +133,7 @@ discover() {
|
||||
# as the rules this harness exists to test. Found 2026-09-13 adding apps/wave-*/tests/. Scoped by
|
||||
# NAME only; manifests_for() then refuses (loudly) any *.test.yaml that is not inside a tests/
|
||||
# directory, so a misplaced file fails rather than disappearing.
|
||||
find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.test.yaml' -print 2>/dev/null
|
||||
find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.test.yaml' -print 2>/dev/null
|
||||
}
|
||||
|
||||
TESTS=()
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
# promtool rule-unit tests for VPNEgressSteeringStale + VPNEgressUplinkLabelMissing
|
||||
# (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
|
||||
# `egress-class=vpn` pods leave through one gateway node's tun0. Cilium resolves that interface to an
|
||||
# IFINDEX and caches it; an OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the
|
||||
# gateway node's up-script publishes `homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex>` so that
|
||||
# node write forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a
|
||||
# destroyed interface and every vpn-class pod is dropped — while tun0 itself stays up, so VPNTunnelDown
|
||||
# says nothing. That shape ran ~80 minutes undetected on the dev gateway.
|
||||
#
|
||||
# VPNEgressSteeringStale detects it by comparing HOW MANY TIMES tun0's ifindex changed in the last hour
|
||||
# against HOW MANY DISTINCT `egress-uplink` label values appeared in the same hour. The subtlety worth
|
||||
# testing is that a label VALUE change is not a sample change: it RETIRES one series and STARTS another,
|
||||
# so "the label followed the tunnel" shows up as two overlapping series in the window, not as a
|
||||
# changing value. Tests 1 and 2 are the same tunnel reconnect with and without that second series, so
|
||||
# they isolate exactly that mechanism — anything that broke the series-counting half would make test 2
|
||||
# fire, and anything that broke the tun0 half would make test 1 quiet.
|
||||
#
|
||||
# VPNEgressUplinkLabelMissing is the WATCHDOG for the rule above: with no uplink label anywhere, the
|
||||
# staleness comparison has nothing to compare and is structurally INERT — present, evaluating, and
|
||||
# incapable of firing. Tests 3 and 4 pin that, because an inert critical alert is the failure mode this
|
||||
# whole group exists to prevent.
|
||||
#
|
||||
# Fixture values are prod's real shape: gateway k3s-prod-3, tun0 ifindex 139 with address
|
||||
# 10.150.192.86 (live label value `10.150.192.86-139`), reconnecting to ifindex 140.
|
||||
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. THE BUG — tun0 was re-created, the label did NOT follow. Steering is stale: FIRES.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect with no label update - VPNEgressSteeringStale fires
|
||||
input_series:
|
||||
# tun0 re-created at t=41m: ifindex 139 -> 140. One change inside the trailing hour.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
# tun0 is UP and passing traffic the whole time — this is what makes the failure deceptive, and
|
||||
# it is asserted below: VPNTunnelDown stays QUIET while egress is actually broken.
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# The node label NEVER changes, so only ONE distinct uplink value exists in the window.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x80'
|
||||
alert_rule_test:
|
||||
# 1 tunnel change > (1 distinct label value - 1) = 0, held for 15m.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare `kubectl get nodes -L
|
||||
homelab.keithjackson.dev/egress-uplink` against `cilium-dbg bpf egress list` on the
|
||||
gateway node; re-applying the CiliumEgressGatewayPolicy (or setting any node label)
|
||||
forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Not yet: at 45m the condition has been true only ~4m, under the 15m `for`.
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
# The watchdog must stay quiet — the label IS present here.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# THE HEADLINE CLAIM, asserted rather than asserted-in-prose: the tunnel alert says NOTHING
|
||||
# while vpn-class egress is completely broken. If VPNTunnelDown covered this failure, the
|
||||
# staleness rule would not need to exist.
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE HEALTHY RECONNECT — same tunnel change, but the label followed it. STAYS QUIET.
|
||||
# This is the half that proves the rule is not simply "any reconnect alerts".
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect WITH the label following - stays quiet
|
||||
input_series:
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# A label VALUE change retires one series and starts another. Both are inside the trailing
|
||||
# hour, so the window holds TWO distinct uplink values: 2 - 1 = 1, which is NOT < 1 change.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x40 _x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-140"}'
|
||||
values: '_x41 1+0x39'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 75m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 3. THE INERT-ALERT CASE — a gateway node publishing NO uplink label at all.
|
||||
# VPNEgressSteeringStale cannot fire in this state, which is exactly why this warning exists.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: gateway node with no egress-uplink label - VPNEgressUplinkLabelMissing fires
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true"}'
|
||||
values: '1+0x60'
|
||||
# tun0 is reconnecting the whole time. The staleness rule STILL cannot fire, because it has no
|
||||
# label series to compare against — that is the inertness being alerted on.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x30 140+0x30'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 35m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without
|
||||
it nothing can tell whether Cilium's egress steering followed the last tunnel reconnect,
|
||||
and `VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Under the 30m `for`.
|
||||
- eval_time: 20m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# And the proof of inertness: the critical rule is silent despite a real reconnect.
|
||||
- eval_time: 55m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 55m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. NO GATEWAY NODE AT ALL — neither rule may fire. Both are guarded by an `and on()` on the
|
||||
# egress-gateway label so they stay silent on a cluster with no VPN gateway, rather than
|
||||
# alerting forever about an absent feature.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: no egress-gateway node - both rules silent
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.10:9100",device="eth0"}'
|
||||
values: '2+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
@@ -24,6 +24,36 @@
|
||||
# pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
|
||||
# its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
|
||||
#
|
||||
# RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds
|
||||
# no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up
|
||||
# cannot protect anything, and ATTEMPTING it actively breaks the backup.
|
||||
#
|
||||
# HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced
|
||||
# mid-flight the PVB fails with `error to expose PVB: error getting client pod <pod>: pods "<pod>"
|
||||
# not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not
|
||||
# advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a
|
||||
# backup that protected every real volume. Any Deployment rollout landing in the backup window is
|
||||
# enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is
|
||||
# how a real backup failure gets ignored, and this one guards R-STATE-03.
|
||||
#
|
||||
# OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a
|
||||
# targeted exclusion:
|
||||
# * ns velero — Velero's OWN transient kopia repository-maintenance Jobs
|
||||
# (`<ns>-default-kopia-maintain-job-<ts>`), emptyDir `plugins` + `scratch`. Four
|
||||
# failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed,
|
||||
# 2 errors, 3079 of 3079 items backed up.
|
||||
# * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs
|
||||
# writable scratch). Four failed PVBs when a runbook edit rolled the Deployment
|
||||
# while the backup was running.
|
||||
#
|
||||
# PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open
|
||||
# question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a
|
||||
# THROWAWAY ConfigMap first, leaving this one untouched until it passed:
|
||||
# control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed
|
||||
# same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
|
||||
# The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted.
|
||||
#
|
||||
|
||||
# `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
|
||||
# narrow it; deliberately not, so a second export is covered too). First matching rule wins.
|
||||
#
|
||||
@@ -44,6 +74,12 @@ data:
|
||||
policies.yaml: |
|
||||
version: v1
|
||||
volumePolicies:
|
||||
# emptyDir first: cheapest condition, and the most common match.
|
||||
- conditions:
|
||||
volumeTypes:
|
||||
- emptyDir
|
||||
action:
|
||||
type: skip
|
||||
- conditions:
|
||||
nfs: {}
|
||||
action:
|
||||
|
||||
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
||||
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
||||
gateway node is the next stop.
|
||||
|
||||
### And now something watches it (2026-09-12)
|
||||
|
||||
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
|
||||
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
|
||||
|
||||
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
|
||||
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
|
||||
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
|
||||
steering pointed at a destroyed ifindex.
|
||||
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
|
||||
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
|
||||
in which the rule above would be inert. The watchdog for the watchdog.
|
||||
|
||||
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
|
||||
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
|
||||
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
|
||||
the label to exist as a metric at all — the two changes must ship together.
|
||||
|
||||
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
|
||||
exposes no metric carrying the egress map's contents (the map is readable only through
|
||||
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
|
||||
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
|
||||
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
|
||||
probe from a vpn-class pod. Neither exists yet.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
||||
|
||||
+14
-4
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
|
||||
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
||||
|
||||
46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
|
||||
46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
|
||||
live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
|
||||
help. These pages are what makes the link resolve.
|
||||
|
||||
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
|
||||
`backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
|
||||
`volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
|
||||
Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
|
||||
direct push is rejected — durable changes go through a branch, a PR, and a green gate.
|
||||
|
||||
## Alerts you should not trust the silence of (measured 2026-09-13)
|
||||
|
||||
@@ -84,8 +85,10 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
|
||||
|
||||
### Networking
|
||||
- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
|
||||
cannot see, and what the kill-switch actually does.
|
||||
- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
|
||||
egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
|
||||
- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
|
||||
ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
|
||||
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
||||
|
||||
### Backups and DR
|
||||
@@ -106,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
|
||||
every green signal missed.
|
||||
|
||||
### Operator procedures (no alert links here)
|
||||
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
|
||||
applies after a `/config` wipe. The claim token is short-lived — read the countdown on
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
|
||||
### Virtualisation
|
||||
- **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
|
||||
- **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
## What this means here
|
||||
|
||||
Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
|
||||
it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
|
||||
it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
|
||||
Kustomization means git and the cluster have diverged and nothing will close the gap.
|
||||
|
||||
The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
|
||||
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
|
||||
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
||||
|
||||
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
||||
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
||||
Where this runbook gives a mutating command it is for an operator, and it says so.
|
||||
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
||||
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
||||
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
||||
and it says so.
|
||||
|
||||
## First commands
|
||||
|
||||
|
||||
Loaded 100 of 123 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user