Compare commits
118
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0c6949ed98 | ||
|
|
20ea70d221 | ||
|
|
8c7a770de2 | ||
|
|
1c5974013e | ||
|
|
4791d4a43a | ||
|
|
dfd608ac32 | ||
|
|
1d2d9d6549 | ||
|
|
4b31f97486 | ||
|
|
50a48e6f07 | ||
|
|
911ead5fbd | ||
|
|
8eb5baffa4 | ||
|
|
4634b44cc7 | ||
|
|
6586919132 | ||
|
|
e6f6e4febb | ||
|
|
e8e6b9c681 | ||
|
|
fcc66c7858 | ||
|
|
b9e8a4f9e8 | ||
|
|
b2d9883958 | ||
|
|
00efd4b828 | ||
|
|
67a54a33a4 | ||
|
|
db1a623073 | ||
|
|
ce271225de | ||
|
|
e72050c690 | ||
|
|
761e25cc3f | ||
|
|
e7508931ca | ||
|
|
bf98a99af6 | ||
|
|
8e017fad91 | ||
|
|
27d0ef3324 | ||
|
|
bbd4f682a6 | ||
|
|
0391139240 | ||
|
|
e9f441b5ce | ||
|
|
234cc6a71a | ||
|
|
69a0d1a1c7 | ||
|
|
d3a0d587a4 | ||
|
|
fe862ab527 | ||
|
|
a2665c0511 | ||
|
|
5a9e455149 | ||
|
|
4fd75ccdaa | ||
|
|
d7f24d0a9e | ||
|
|
712f4c9336 | ||
|
|
e3e56af461 | ||
|
|
a3f4ddb971 | ||
|
|
5f3ea63b7f | ||
|
|
23c335257b | ||
|
|
9339d5a79e | ||
|
|
a1cc0a5556 | ||
|
|
6eaaaae334 | ||
|
|
cde02d19d2 | ||
|
|
05a8094785 | ||
|
|
d99d47494c | ||
|
|
6f231a4311 | ||
|
|
623fa14256 | ||
|
|
16cbd4a79e | ||
|
|
755c08ab40 | ||
|
|
b389566641 | ||
|
|
7887ecbbdc | ||
|
|
d717d8e979 | ||
|
|
bc106caac1 | ||
|
|
79c703bc79 | ||
|
|
3523617671 | ||
|
|
6a4dc0ada7 | ||
|
|
14d5ccbadd | ||
|
|
05881d5a6a | ||
|
|
cf0d6c6beb | ||
|
|
20a28d0993 | ||
|
|
b184a42729 | ||
|
|
3173254cc4 | ||
|
|
7cb209b9a0 | ||
|
|
12d4e0ae40 | ||
|
|
989781b352 | ||
|
|
97257515e8 | ||
|
|
b4bc8e1cb9 | ||
|
|
83f13f4b0a | ||
|
|
adda6dd779 | ||
|
|
4664b41db3 | ||
|
|
4545611f4d | ||
|
|
fd43f332d6 | ||
|
|
4080065aa7 | ||
|
|
ce7819a7da | ||
|
|
3de79f81a5 | ||
|
|
5c96f4f3a7 | ||
|
|
79d9ab2c8c | ||
|
|
de861b8672 | ||
|
|
207c939f89 | ||
|
|
aae63c97a3 | ||
|
|
3a72b07cda | ||
|
|
b9f3143f42 | ||
|
|
7369495b67 | ||
|
|
dafa9cd241 | ||
|
|
0841e2461c | ||
|
|
8891df07b0 | ||
|
|
268b0a26b4 | ||
|
|
4f2ca8c1a3 | ||
|
|
bd17b65db9 | ||
|
|
24b8111d0c | ||
|
|
229d7f2d9c | ||
|
|
a62242bd78 | ||
|
|
bfb4922de1 | ||
|
|
f663e7c801 | ||
|
|
d96165e8fb | ||
|
|
52398d4426 | ||
|
|
4687521d65 | ||
|
|
69a99e6ca6 | ||
|
|
eb2510837c | ||
|
|
cf7d97e546 | ||
|
|
cd454886c7 | ||
|
|
72f0ba4e1f | ||
|
|
088ff13d15 | ||
|
|
cc648ba02c | ||
|
|
8b36242643 | ||
|
|
e4bd76070e | ||
|
|
aff80d1d7a | ||
|
|
a76ae3bdfa | ||
|
|
02c58aae15 | ||
|
|
f57c411953 | ||
|
|
0fe12b8332 | ||
|
|
204f0ec28e | ||
|
|
16c6234696 |
No files matched your search
+28
-14
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
|
||||
*collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
|
||||
(`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
|
||||
|
||||
## Honest SKIP — never fake green (Phase 0)
|
||||
## Honest SKIP — never fake green
|
||||
|
||||
**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
|
||||
exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
|
||||
skip logic below is still correct and still honest, but read it as describing a **live
|
||||
coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
|
||||
phrased as a phase is part of why the gap went unexamined.
|
||||
|
||||
Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
|
||||
they do not pass**:
|
||||
|
||||
- The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
|
||||
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
|
||||
## Triggers & branch model (CONTRIBUTING.md)
|
||||
|
||||
- `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
|
||||
- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
|
||||
- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
|
||||
so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
|
||||
**No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
|
||||
protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
|
||||
|
||||
## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
|
||||
## Transition: COMPLETE (2026-09-29)
|
||||
|
||||
Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
|
||||
[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
|
||||
This section used to describe a migration as pending. It has happened. Gitea is live at
|
||||
`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
|
||||
`gate.yml` runs on every PR.
|
||||
|
||||
1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
|
||||
but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
|
||||
local feature-branch → `main`, the gate enforced by convention + the local runner.
|
||||
2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
|
||||
3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
|
||||
enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
|
||||
**platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
|
||||
**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
|
||||
anyone checking the platform. In reality `main` had **no branch protection** and **zero
|
||||
runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
|
||||
between rested on an assertion nobody verified. PR #1 — the first pull request this
|
||||
repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
|
||||
|
||||
So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
|
||||
Query the API. `infra/gitea/` holds the scripts and the measured state.
|
||||
|
||||
Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
|
||||
CI. That is a real coverage gap, not a phase.
|
||||
|
||||
## Configuration (set in Gitea; referenced by name only)
|
||||
|
||||
|
||||
@@ -44,16 +44,31 @@
|
||||
---
|
||||
name: gate
|
||||
|
||||
# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
|
||||
#
|
||||
# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
|
||||
# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
|
||||
#
|
||||
# * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
|
||||
# SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
|
||||
# clock on the one result that actually gates the merge. That cost real time during the
|
||||
# 2026-09-29 plex claim, where the loop is racing a short-lived token.
|
||||
# * The list silently did not match this repo's own convention. Branches here are `feat/**`,
|
||||
# which was absent, so the "fast feedback" never fired for the most common prefix anyway —
|
||||
# a mismatch nobody noticed for a month because its only symptom was a MISSING run.
|
||||
# Any hand-maintained prefix list has that failure mode; deleting it removes the class.
|
||||
# * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
|
||||
# you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
|
||||
# same checks buys nothing.
|
||||
#
|
||||
# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
|
||||
# which no PR context exists.
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- 'feature/**'
|
||||
- 'fix/**'
|
||||
- 'docs/**'
|
||||
- 'chore/**'
|
||||
|
||||
concurrency:
|
||||
group: gate-${{ github.ref }}
|
||||
@@ -95,9 +110,31 @@ jobs:
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
# GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
|
||||
# OWN database (branch protection + runner registration) — the blind spot that let `main`
|
||||
# sit unprotected with zero runners for a month while this repo recorded both as working.
|
||||
# Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
|
||||
#
|
||||
# WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
|
||||
# (spec-immutability). It now also covers gitea-settings skipping when that secret is
|
||||
# absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
|
||||
# every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
|
||||
# exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
|
||||
# tolerance narrows back to spec-immutability alone.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
|
||||
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
|
||||
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
|
||||
# skipping, this line turns that into a failure and the PR goes red instead of
|
||||
# merging. Green here means the token resolved and the live Gitea matched the
|
||||
# declaration in infra/gitea/expected-branch-protection.json.
|
||||
REQUIRE_GITEA_SETTINGS: "true"
|
||||
# ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
|
||||
# PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
|
||||
GITHUB_EVENT_NAME: ${{ github.event_name }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
|
||||
@@ -11,11 +11,14 @@
|
||||
# v
|
||||
# merge-gate decision [R-TEST-03: touched subset green]
|
||||
#
|
||||
# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
|
||||
# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
|
||||
# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
|
||||
# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
|
||||
# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
|
||||
# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
|
||||
# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
|
||||
# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
|
||||
# as a plan and never revisited once the plan completed.
|
||||
#
|
||||
# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
|
||||
# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
|
||||
# statement about the live platform; query the API instead (see infra/gitea/).
|
||||
#
|
||||
# CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
|
||||
# scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
|
||||
# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
|
||||
# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
|
||||
# merely left alone. That is a one-way, outward-facing operation against a live authoritative
|
||||
# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
|
||||
# reconcile removes can be restored by hand from this file. It is the same discipline as
|
||||
# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
|
||||
# change rather than to a migration.
|
||||
#
|
||||
# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
|
||||
# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
|
||||
# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
|
||||
# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
|
||||
# devcontainer, which has previously produced false "no records" readings — the API is the only
|
||||
# reading that is both complete and authoritative.
|
||||
#
|
||||
# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
|
||||
# TTL and the proxy flag, not just the value.
|
||||
#
|
||||
# <type>|<name>|<content>|<ttl>|<proxied>
|
||||
#
|
||||
# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
|
||||
# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
|
||||
#
|
||||
# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
|
||||
# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
|
||||
# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
|
||||
# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
|
||||
# cannot touch them regardless of what this branch does:
|
||||
#
|
||||
# *.keithjackson.dev -> 136.51.115.252 legacy stack (the zone wildcard)
|
||||
# brickbreak.keithjackson.dev -> 20.236.242.14 unrelated, off-prem
|
||||
# dev.brickbreak… -> 20.236.242.14 unrelated, off-prem
|
||||
#
|
||||
# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
|
||||
# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
|
||||
# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
|
||||
# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
|
||||
# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
|
||||
# end state for `internal` services and it is also precisely why Pi-hole must already be
|
||||
# answering the LAN authoritatively before this branch merges.
|
||||
#
|
||||
# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
|
||||
# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
|
||||
# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
|
||||
# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
|
||||
# HTTPRoute that already had a published record produced
|
||||
# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
|
||||
# interval (~60s) later, and nothing else in the zone moved.
|
||||
#
|
||||
# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
|
||||
# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
|
||||
# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
|
||||
# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
|
||||
# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
|
||||
# points the registry at a source object that does not exist.
|
||||
A|*.keithjackson.dev|136.51.115.252|1|false
|
||||
A|auth.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
|
||||
A|crafty.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|gitea.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|grafana.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|jackett.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|pihole.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|plex.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|radarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|seerr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|unifi.home.keithjackson.dev|192.168.2.21|1|false
|
||||
TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
|
||||
TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
|
||||
TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
|
||||
TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
|
||||
TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
|
||||
TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
|
||||
TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
|
||||
TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
|
||||
TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
|
||||
TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
|
||||
TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
|
||||
TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
|
||||
TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
|
||||
TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
|
||||
TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
|
||||
TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false
|
||||
@@ -0,0 +1,168 @@
|
||||
# CLAUDE.md — how work lands in this repo
|
||||
|
||||
Read this before doing anything durable. It is the delivery contract for AI coding
|
||||
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
|
||||
Every durable change reaches `main` exactly one way:
|
||||
|
||||
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
|
||||
|
||||
There is no other path. Specifically:
|
||||
|
||||
- **Do not `git push origin main`.**
|
||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||
- **Do not merge your own PR.** See the approval model below — and either way, the merge is
|
||||
someone else's action, never yours.
|
||||
|
||||
### Who approves
|
||||
|
||||
**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
|
||||
branch requires no human approval for one. That is a deliberate transfer of authority, on the
|
||||
operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
|
||||
good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
|
||||
add a human.
|
||||
|
||||
**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
|
||||
admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
|
||||
approving review from someone who is not its author. The reason is narrow and structural — **a
|
||||
change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
|
||||
the weakened check. Everything else merges on green.
|
||||
|
||||
The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
|
||||
sensitive path, so the policy cannot be widened or narrowed without a human.
|
||||
|
||||
That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
|
||||
missing credential is a lock-out, here a skip would be a bypass.
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
|
||||
> That wording is **stale** — it predates the PR flow. It describes what Flux does
|
||||
> (main is what the cluster follows), not what you are permitted to do. Honour this
|
||||
> file, not that line. Fixing those runbooks is tracked work.
|
||||
|
||||
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
|
||||
|
||||
Verified against the live Gitea on 2026-09-29:
|
||||
|
||||
| Expected | Actual |
|
||||
|---|---|
|
||||
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
|
||||
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
|
||||
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
|
||||
| Approval required before merge | ✅ `required_approvals: 1` |
|
||||
|
||||
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
true. They are true now because each was measured, and the push rejection was provoked rather
|
||||
than assumed. **Re-measure before relying on any of them** — see §2b.
|
||||
|
||||
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
|
||||
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
|
||||
|
||||
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||
|
||||
## 2b. Never infer platform state from this repository
|
||||
|
||||
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
|
||||
Both describe what someone believed at the time of writing, and both have been
|
||||
confidently wrong for a month at a stretch.
|
||||
|
||||
On 2026-09-29 the log stated the merge gate was a required status check with a runner
|
||||
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
|
||||
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
|
||||
was wrong, and nothing in the tree could have revealed it, because the tree was the source
|
||||
of the error.
|
||||
|
||||
**So, before asserting that any gate, protection, runner, secret, or workload is in a
|
||||
given state: query the thing itself.**
|
||||
|
||||
| Claim about... | Check |
|
||||
|---|---|
|
||||
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
|
||||
| a runner exists | `GET /api/v1/admin/actions/runners` |
|
||||
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
|
||||
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
|
||||
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
|
||||
|
||||
Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
|
||||
"unverified" is a legitimate report, and a far better one than a confident guess that
|
||||
matches the docs.
|
||||
|
||||
**A green tick is not proof anything ran.** Read what actually executed.
|
||||
|
||||
## 3. Gate before you push
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
|
||||
Exit codes are a contract — do not flatten them:
|
||||
|
||||
| rc | Meaning |
|
||||
|---|---|
|
||||
| 0 | PASS |
|
||||
| 1 | BLOCKED — something failed |
|
||||
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
|
||||
|
||||
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
|
||||
beside this one; that single skip is expected and tolerated. Any *other* skip is a
|
||||
real gap in your run.
|
||||
|
||||
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
|
||||
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
|
||||
|
||||
### Running the gate from a git worktree
|
||||
|
||||
Two unrelated failures, both with known workarounds — put both in any agent brief:
|
||||
|
||||
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
|
||||
and exists only in the main checkout. Do **not** copy it (it is a credential).
|
||||
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
|
||||
normally.
|
||||
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
|
||||
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
|
||||
the `ENGINE` hook — a shim that intercepts `run` and re-execs
|
||||
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
|
||||
`tracked_files()` is this bug, not a broken tree.
|
||||
|
||||
## 4. Branch naming
|
||||
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
|
||||
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
|
||||
|
||||
## 5. Cluster access
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the
|
||||
pinned container. Always name the context explicitly:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
|
||||
default has already changed once. Never rely on it. Do not run two `dev.sh`
|
||||
invocations concurrently.
|
||||
|
||||
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
|
||||
|
||||
## 6. Secrets
|
||||
|
||||
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
|
||||
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
|
||||
list key names or object names only. A password has been leaked into a transcript
|
||||
this way before.
|
||||
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
|
||||
the single creation-rules file; do not add a second one in a subtree.
|
||||
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
|
||||
flow. Do not design around committing one — read the expiry from the issuing page and
|
||||
treat a burnt token as free to retry.
|
||||
|
||||
## 7. Before you finish
|
||||
|
||||
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
|
||||
verified, and what you did **not** verify. State gaps plainly — an honest
|
||||
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
|
||||
burned by the difference.
|
||||
+2067
File diff suppressed because it is too large.
Load diff
@@ -38,22 +38,56 @@ ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render
|
||||
|
||||
## Status
|
||||
|
||||
Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
|
||||
open** — `clusters/prod` is a design-only overlay.
|
||||
Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
|
||||
progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
|
||||
apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
|
||||
dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
|
||||
hardware rather than on Proxmox VMs.
|
||||
|
||||
**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
|
||||
alerts live*:
|
||||
|
||||
| criterion | state |
|
||||
|---|---|
|
||||
| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
|
||||
| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
|
||||
| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
|
||||
|
||||
**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
|
||||
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
|
||||
`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
|
||||
and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
|
||||
class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
|
||||
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
|
||||
*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
|
||||
|
||||
**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
|
||||
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
|
||||
exercise rather than something to run in passing.
|
||||
|
||||
Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
|
||||
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
|
||||
|
||||
| Phase | Delivered |
|
||||
|---|---|
|
||||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||||
| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
|
||||
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
|
||||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||||
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
|
||||
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
|
||||
|
||||
**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
|
||||
(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
|
||||
the full report is `conformance/_artifacts/coverage/coverage.md`.
|
||||
**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
|
||||
2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
|
||||
`./conformance/run.sh --coverage-only`; the full report is
|
||||
`conformance/_artifacts/coverage/coverage.md`.
|
||||
|
||||
Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
|
||||
*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
|
||||
SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
|
||||
second.
|
||||
|
||||
The running narrative — decisions, root causes, incidents, follow-ups — is
|
||||
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
|
||||
|
||||
@@ -41,10 +41,41 @@ spec:
|
||||
# disk KubeVirt cannot migrate and eviction would hard-stop the guest.
|
||||
evictionStrategy: LiveMigrate
|
||||
domain:
|
||||
# =====================================================================================
|
||||
# FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
|
||||
# =====================================================================================
|
||||
# MEASURED on prod 2026-09-14, not assumed:
|
||||
# * the NoCloud seed KubeVirt builds for this VM carries
|
||||
# {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
|
||||
# (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
|
||||
# EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
|
||||
# had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
|
||||
# was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
|
||||
# standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
|
||||
# * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
|
||||
# cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
|
||||
#
|
||||
# CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
|
||||
# worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
|
||||
# unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
|
||||
# every per-instance module (write_files, runcmd). The new payload is silently ignored.
|
||||
#
|
||||
# So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
|
||||
# process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
|
||||
# it also makes a rebuild reproducible — a fresh apply would otherwise take a random
|
||||
# webhook-generated UUID and give the guest a different identity every time.
|
||||
#
|
||||
# The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
|
||||
# difference is what makes the next boot a NEW cloud-init instance and installs the
|
||||
# `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
|
||||
# already stamped, so this change does not also churn the guest's DMI serial.
|
||||
firmware:
|
||||
uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
|
||||
serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
|
||||
# FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
|
||||
# so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
|
||||
# is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
|
||||
# add: firmware: { bootloader: { efi: { secureBoot: false } } }
|
||||
# is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
|
||||
# the guest reports Ubuntu 24.04.4 LTS.
|
||||
cpu:
|
||||
cores: 2
|
||||
memory:
|
||||
@@ -58,9 +89,28 @@ spec:
|
||||
- name: cloudinit
|
||||
disk:
|
||||
bus: virtio
|
||||
# The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
|
||||
# iso9660 disk — see the `uosdata` volume near the bottom of this file.
|
||||
- name: uosdata
|
||||
disk:
|
||||
bus: virtio
|
||||
interfaces:
|
||||
# MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
|
||||
# KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
|
||||
# /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
|
||||
# that file while the NoCloud instance-id is unchanged — and the instance-id here is
|
||||
# domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
|
||||
# handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
|
||||
# administratively DOWN with no address and no DHCP attempt.
|
||||
# That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
|
||||
# device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
|
||||
# (because the per-instance modules were skipped), yet UOS is installed and
|
||||
# qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
|
||||
# Pinning the MAC to the value the current VMI already holds makes the guest's netplan
|
||||
# match permanently, so this cannot recur on the next recreation.
|
||||
- name: default
|
||||
masquerade: {}
|
||||
macAddress: a6:d7:c0:b3:72:fa
|
||||
# Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
|
||||
# these and (b) surfaces them as containerPorts on the virt-launcher pod so the
|
||||
# LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
|
||||
@@ -83,41 +133,112 @@ spec:
|
||||
- name: rootdisk
|
||||
dataVolume:
|
||||
name: unifi-rootdisk
|
||||
# =====================================================================================
|
||||
# THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
|
||||
# =====================================================================================
|
||||
# KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
|
||||
# each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
|
||||
# reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
|
||||
# REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
|
||||
# URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
|
||||
# operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
|
||||
#
|
||||
# `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
|
||||
# Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
|
||||
# and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
|
||||
# ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
|
||||
# guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
|
||||
# only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
|
||||
# works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
|
||||
# read-only console output with no virtctl at all.
|
||||
#
|
||||
# WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
|
||||
# 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
|
||||
# concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
|
||||
# L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
|
||||
# envsubst over the DECRYPTED resources, so the escape contract in
|
||||
# clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
|
||||
# shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
|
||||
# ONLY the operator's two values behind SOPS keeps both properties.
|
||||
#
|
||||
# A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
|
||||
# disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
|
||||
# re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
|
||||
# the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
|
||||
- name: uosdata
|
||||
secret:
|
||||
secretName: unifi-uos-installer
|
||||
volumeLabel: UOSDATA
|
||||
- name: cloudinit
|
||||
cloudInitNoCloud:
|
||||
# INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
|
||||
# userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
|
||||
# signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
|
||||
# Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
|
||||
# NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
|
||||
# ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
|
||||
# Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
|
||||
# as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
|
||||
# by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
|
||||
# an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
|
||||
# and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
|
||||
# flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
|
||||
# text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
|
||||
# wording in clusters/base/apps-vms.yaml overstated it.
|
||||
# INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
|
||||
# DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
|
||||
# otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
|
||||
# "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
|
||||
# larger data." The payload below is ~1.5 KB; keep additions small.
|
||||
#
|
||||
# WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
|
||||
# also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
|
||||
# plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
|
||||
# makes a later URL change take effect, because cloud-init will not re-run per-instance
|
||||
# modules while the instance-id is unchanged.
|
||||
#
|
||||
# `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
|
||||
# out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
|
||||
# the serial console log for the current boot shows both `uosserver.service` and
|
||||
# `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
|
||||
# UOS is installed here; see the README for what is actually wrong (the guest lost its
|
||||
# IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
|
||||
# where the unit name changes in some future UOS release.
|
||||
#
|
||||
# ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
|
||||
# 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
|
||||
# is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
|
||||
# which envsubst leaves untouched — verified against the live VMI, whose rendered
|
||||
# userData still carries its shell variables verbatim. There is deliberately NO
|
||||
# brace-form shell expansion left in this payload, so nothing here needs escaping; if
|
||||
# you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
|
||||
# write an indexed expansion.
|
||||
userData: |-
|
||||
#cloud-config
|
||||
hostname: unifi
|
||||
fqdn: unifi.${BASE_DOMAIN}
|
||||
package_update: true
|
||||
package_upgrade: false
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
|
||||
write_files:
|
||||
- path: /usr/local/sbin/install-uos
|
||||
permissions: '0755'
|
||||
content: |
|
||||
#!/usr/bin/env bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
if [ -z "$${UOS_URL:-}" ]; then
|
||||
UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
|
||||
fi
|
||||
[ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
|
||||
t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
|
||||
d=/run/uos; mkdir -p $d
|
||||
mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
|
||||
p=$(tr -d '\r\n' < $d/console-password || true)
|
||||
case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
|
||||
systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
|
||||
s=/var/lib/uos-installed
|
||||
[ -e $s ] && exit 0
|
||||
u=$(tr -d '\r\n' < $d/installer-url || true)
|
||||
case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
|
||||
t=$(mktemp)
|
||||
curl -fL --retry 5 --retry-delay 10 -o $t "$u"
|
||||
chmod +x $t
|
||||
yes y | $t install || yes y | $t
|
||||
rm -f $t; date -Is > $s
|
||||
- path: /etc/systemd/system/uos-install.service
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Install UniFi OS Server
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/install-uos
|
||||
StandardOutput=journal+console
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
runcmd:
|
||||
- [ systemctl, enable, --now, qemu-guest-agent ]
|
||||
- [ systemctl, enable, --now, podman.socket ]
|
||||
- [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
|
||||
- [systemctl, enable, --now, qemu-guest-agent]
|
||||
- [systemctl, enable, --now, podman.socket]
|
||||
- [systemctl, enable, --now, uos-install.service]
|
||||
@@ -21,8 +21,61 @@
|
||||
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
|
||||
#
|
||||
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
|
||||
# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does).
|
||||
# into the guest.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
|
||||
# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
|
||||
#
|
||||
# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
|
||||
# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
|
||||
# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
|
||||
# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
|
||||
# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
|
||||
# announcing node that holds no backend has nowhere to send the packet and answers ICMP port
|
||||
# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
|
||||
# (the leases were 8 days old); the operator's controller UI and device inform path were down.
|
||||
# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
|
||||
# deleting the Lease forces a re-election — and that re-election is random, so it can pick a
|
||||
# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
|
||||
# reference implementation this file now follows.
|
||||
#
|
||||
# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
|
||||
# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
|
||||
# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
|
||||
# controller has always seen the pod-side masquerade gateway as the source, never the LAN
|
||||
# client. There is a SECOND translation below the Service, and it already destroyed the very
|
||||
# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
|
||||
# it is giving up an address the guest could not see in the first place.
|
||||
#
|
||||
# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
|
||||
# MAC (and the device reports its own IP inside the inform payload), not the packet source,
|
||||
# and guest-portal authorization runs on the gateway appliance, not on this controller.
|
||||
#
|
||||
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
|
||||
# is what makes this survive a reschedule and a rebuild.
|
||||
#
|
||||
# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
|
||||
# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
|
||||
# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
||||
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
||||
# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
|
||||
# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
|
||||
# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
||||
# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
|
||||
# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
||||
#
|
||||
# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
|
||||
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
|
||||
# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
|
||||
# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
|
||||
# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
|
||||
# fix would be a NEW restriction that could silently break a remote site's device informing over
|
||||
# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
|
||||
# change, made deliberately and tested against the device fleet.
|
||||
#
|
||||
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
||||
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
||||
@@ -49,7 +102,7 @@ metadata:
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
externalTrafficPolicy: Cluster
|
||||
selector:
|
||||
kubevirt.io/vm: unifi
|
||||
ports:
|
||||
@@ -72,7 +125,7 @@ metadata:
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
externalTrafficPolicy: Cluster
|
||||
selector:
|
||||
kubevirt.io/vm: unifi
|
||||
ports:
|
||||
|
||||
+202
-74
@@ -1,18 +1,48 @@
|
||||
# UniFi Network controller (UniFi OS Server) — VM onboarding
|
||||
|
||||
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
|
||||
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
|
||||
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
|
||||
> or install cannot stall the app tier.
|
||||
> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
|
||||
> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
|
||||
> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
|
||||
> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
|
||||
> cannot stall the app tier.
|
||||
>
|
||||
> **Status (verified 2026-08-26):**
|
||||
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
|
||||
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
|
||||
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
|
||||
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
|
||||
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
|
||||
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
|
||||
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
|
||||
> **Status (re-verified against live prod 2026-09-14):**
|
||||
> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
|
||||
> - CDI import — **Succeeded** (100%).
|
||||
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
|
||||
> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
|
||||
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
|
||||
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
|
||||
> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
|
||||
> `:8080` refuse connections.
|
||||
|
||||
## What is actually wrong (2026-09-14)
|
||||
|
||||
The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
|
||||
That was wrong, and acting on it would have meant reinstalling over a working controller.
|
||||
|
||||
Evidence, all read-only, from the live cluster:
|
||||
|
||||
| Claim | How it was checked | Result |
|
||||
|---|---|---|
|
||||
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
|
||||
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
|
||||
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
|
||||
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
|
||||
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
|
||||
|
||||
So the controller is installed and its units start; nothing downstream of the guest's own network
|
||||
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
|
||||
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
|
||||
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
|
||||
guest.** Diagnosing it needs a shell, which is the second problem below.
|
||||
|
||||
**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
|
||||
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
|
||||
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
|
||||
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
|
||||
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
|
||||
new Secret is for.
|
||||
|
||||
## What this is
|
||||
|
||||
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
|
||||
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
|
||||
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
|
||||
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
|
||||
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
|
||||
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
|
||||
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
|
||||
| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
|
||||
| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
|
||||
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
|
||||
|
||||
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
|
||||
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
|
||||
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
|
||||
`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
|
||||
version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
|
||||
### Cloud-init delivery — two constraints learned the hard way
|
||||
### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
|
||||
|
||||
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
|
||||
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
|
||||
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
|
||||
Secret deliberately.
|
||||
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
|
||||
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
|
||||
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
|
||||
1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
|
||||
of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
|
||||
userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
|
||||
payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
|
||||
is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
|
||||
validate as an unknown field.
|
||||
2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
|
||||
nothing.** The NoCloud seed's `instance-id` is exactly
|
||||
`spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
|
||||
(so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
|
||||
product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
|
||||
`runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
|
||||
effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
|
||||
to force a re-run. See the long comment in `30-virtualmachine.yaml`.
|
||||
3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
|
||||
been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
|
||||
envsubst therefore runs over the cloud-init payload, and the escape contract in
|
||||
`clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
|
||||
against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
|
||||
indexed expansion is a hard error that fails the whole Kustomization.
|
||||
|
||||
### Node prerequisites this VM forced onto the cluster
|
||||
|
||||
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
|
||||
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
|
||||
3-replica class made the import fail with insufficient storage.
|
||||
|
||||
## UOS install method (researched)
|
||||
## UOS install method (researched 2026-09-14)
|
||||
|
||||
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
|
||||
2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
|
||||
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
|
||||
`ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
|
||||
**`https://<host>:11443`**.
|
||||
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
|
||||
independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
|
||||
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
|
||||
Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
|
||||
(the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
|
||||
Podman must be at `/usr/bin/podman`.
|
||||
- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
|
||||
in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
|
||||
- **Is the URL a short-lived signed link?** *Probably not.* The links look like
|
||||
`https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
|
||||
**no signature query string**, and public community installers hard-code them. It is
|
||||
account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
|
||||
account-derived material out of plaintext config — but do **not** build logic around expiry.
|
||||
- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
|
||||
Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
|
||||
ecosystem actually does is pipe confirmations in: the widely used community installer
|
||||
(`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
|
||||
either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
|
||||
`yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
|
||||
fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
|
||||
(no URL has been supplied, and UOS is already installed here), so treat it as "established by
|
||||
convergent third-party practice", not as verified-by-us.
|
||||
- The previously shipped helper had two defects worth recording: it resolved the URL from
|
||||
`fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
|
||||
`unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
|
||||
|
||||
### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
|
||||
|
||||
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
|
||||
| 3478 | UDP | **STUN** (device ⇄ controller) |
|
||||
| 10001 | UDP | **L2 device discovery** |
|
||||
|
||||
**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
|
||||
the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
|
||||
|
||||
## L4 exposure design (the hard part)
|
||||
|
||||
The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
|
||||
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
|
||||
|
||||
- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
|
||||
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
|
||||
The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
|
||||
flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
|
||||
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
|
||||
(`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
|
||||
`default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
|
||||
the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
|
||||
existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
|
||||
/ `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
|
||||
decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
|
||||
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
|
||||
declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
|
||||
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
|
||||
— via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
|
||||
Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
|
||||
reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
|
||||
masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
|
||||
explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
|
||||
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
|
||||
LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
|
||||
`minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
|
||||
declared ports into the guest.
|
||||
- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
|
||||
used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
|
||||
Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
|
||||
leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
|
||||
and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
|
||||
it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
|
||||
why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
|
||||
- **Device-adoption story:** point devices' **inform host** at the VIP
|
||||
(`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
|
||||
`set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
|
||||
subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
|
||||
broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
|
||||
zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
|
||||
on other VLANs must route to the VIP and be inform-pointed anyway.
|
||||
- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
|
||||
per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
|
||||
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
|
||||
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
|
||||
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
|
||||
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
|
||||
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
|
||||
flip, so this hostname is current, not a dev leftover.
|
||||
|
||||
## The operator's one field — installing UOS without a shell
|
||||
|
||||
`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
|
||||
the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
|
||||
`uos-install.service` reads it **on every boot**:
|
||||
|
||||
```bash
|
||||
sops apps/vms/unifi/unifi-uos-installer.sops.yaml # fill installer-url and/or console-password
|
||||
git commit && git push # merge to the branch prod's Flux tracks
|
||||
kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
|
||||
kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
|
||||
-c guest-console-log # watch the boot + the install, read-only
|
||||
```
|
||||
|
||||
Design notes that matter if you change this:
|
||||
|
||||
- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
|
||||
cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
|
||||
L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
|
||||
resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
|
||||
operator values in SOPS.
|
||||
- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
|
||||
the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
|
||||
firmware-UUID bump.
|
||||
- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
|
||||
a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
|
||||
guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
|
||||
controller on next boot".
|
||||
- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
|
||||
`apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
|
||||
`clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
|
||||
guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
|
||||
|
||||
## Open questions / risks
|
||||
|
||||
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
|
||||
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
|
||||
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
|
||||
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
|
||||
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
|
||||
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
|
||||
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
|
||||
1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
|
||||
requires a shell, which requires `console-password` above plus one restart. First things to run
|
||||
on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
|
||||
`ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
|
||||
own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
|
||||
capture the state before rebooting a second time if you can.
|
||||
2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
|
||||
install-method section: no official silent flag exists, `yes y | <installer> install` is the
|
||||
convergent community practice, and it has not been executed on our guest. Treat the first real
|
||||
install as something to watch on the console log, not as a fire-and-forget.
|
||||
3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
|
||||
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
|
||||
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
|
||||
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
|
||||
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
|
||||
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
|
||||
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
|
||||
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
|
||||
4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
|
||||
5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
|
||||
prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
|
||||
(`.23`) or the node addresses (`.10/.11/.12`).
|
||||
6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
|
||||
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
|
||||
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
|
||||
7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
|
||||
installer and for optional remote access.
|
||||
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
|
||||
8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
|
||||
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
|
||||
9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
|
||||
`kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
|
||||
`virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
|
||||
the cluster's **v1.9.0**.
|
||||
|
||||
## Remaining bring-up
|
||||
|
||||
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
|
||||
Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
|
||||
is written for a from-scratch cluster; the prod addresses are the live ones.
|
||||
|
||||
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
|
||||
1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
|
||||
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
|
||||
Kustomization (not an app wave).
|
||||
Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
|
||||
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
|
||||
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
|
||||
`192.168.1.54`.
|
||||
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
|
||||
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
|
||||
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
|
||||
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
|
||||
`192.168.2.21`.
|
||||
5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
|
||||
`uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
|
||||
fill `installer-url`, commit, and the guest installs itself on first boot.
|
||||
6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
|
||||
console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
|
||||
7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
|
||||
complete the first-run wizard.
|
||||
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
|
||||
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
|
||||
8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
|
||||
split-horizon) and from in-cluster (coredns-custom).
|
||||
9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
|
||||
connects (green). Test guest portal / throughput if used.
|
||||
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
|
||||
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
|
||||
10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
|
||||
11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
|
||||
|
||||
**Quick health check today:**
|
||||
```bash
|
||||
kubectl -n unifi get dv,vmi,svc
|
||||
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
|
||||
# the two that would have caught the current failure:
|
||||
kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}' # expect an ipAddress
|
||||
kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
|
||||
-c compute -- ip neigh # expect 10.0.2.2 REACHABLE
|
||||
```
|
||||
@@ -11,3 +11,11 @@ resources:
|
||||
- 30-virtualmachine.yaml
|
||||
- 40-networkpolicy.yaml
|
||||
- 50-service-l4.yaml
|
||||
# SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
|
||||
# into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
|
||||
# Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
|
||||
# FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
|
||||
# template + the operator hand-off steps live in the `.example` sibling of the file below, which
|
||||
# is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
|
||||
# text contains an example template's filename, on the assumption it is being applied.
|
||||
- unifi-uos-installer.sops.yaml
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: unifi-uos-installer
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
|
||||
console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
|
||||
WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
|
||||
TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
|
||||
cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
|
||||
kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-19T02:47:13Z"
|
||||
mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -0,0 +1,71 @@
|
||||
# =============================================================================
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
|
||||
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
|
||||
# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
|
||||
# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
|
||||
# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
|
||||
# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
|
||||
#
|
||||
# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
|
||||
# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
|
||||
# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
|
||||
# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
|
||||
# working controller. `console-password` IS useful today: it is the only way to get a shell.
|
||||
#
|
||||
# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
|
||||
# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
|
||||
# ours to redistribute. Treat it as account-derived material, not as public configuration.
|
||||
# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
|
||||
# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
|
||||
# and public community install scripts hard-code these URLs — so it does not behave like a
|
||||
# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
|
||||
# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
|
||||
#
|
||||
# HAND-OFF STEPS (operator) — this is the whole job:
|
||||
# (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
|
||||
# copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
|
||||
# fetches it itself.
|
||||
# (b) Open the encrypted sibling in your editor:
|
||||
# sops apps/vms/unifi/unifi-uos-installer.sops.yaml
|
||||
# (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
|
||||
# placeholder rather than failing.
|
||||
# (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
|
||||
# (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
|
||||
# kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
|
||||
# Watch it come up WITHOUT virtctl:
|
||||
# kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
|
||||
# -o name | head -1) -c guest-console-log
|
||||
# (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
|
||||
#
|
||||
# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
|
||||
# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
|
||||
# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
|
||||
# payload ITSELF changes.
|
||||
#
|
||||
# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
|
||||
# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: unifi-uos-installer
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
# KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
|
||||
# /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
|
||||
#
|
||||
# The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
|
||||
# (this placeholder included) is treated by the guest as "not provided yet": the unit logs
|
||||
# "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
|
||||
installer-url: REPLACE-ME
|
||||
# Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
|
||||
# THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
|
||||
# no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
|
||||
# only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
|
||||
# virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
|
||||
# openssl rand -base64 24
|
||||
# Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
|
||||
console-password: REPLACE-ME
|
||||
+2
-22
@@ -238,26 +238,6 @@ spec:
|
||||
- name: gitea
|
||||
port: 3000
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: gitea
|
||||
namespace: gitea
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: gitea
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- gitea
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -272,11 +252,11 @@ spec:
|
||||
- name: gitea.slis
|
||||
rules:
|
||||
- alert: gitea-down
|
||||
expr: up{job="gitea"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: gitea
|
||||
annotations:
|
||||
summary: gitea is down
|
||||
description: 'gitea has been unreachable for 5m (SLI: up).'
|
||||
description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -268,6 +268,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -284,26 +285,6 @@ spec:
|
||||
- name: jackett
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: jackett
|
||||
namespace: jackett
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: jackett
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jackett
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- jackett
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -318,11 +299,11 @@ spec:
|
||||
- name: jackett.slis
|
||||
rules:
|
||||
- alert: jackett-down
|
||||
expr: up{job="jackett"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: jackett
|
||||
annotations:
|
||||
summary: jackett is down
|
||||
description: 'jackett has been unreachable for 5m (SLI: up).'
|
||||
description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -14,9 +14,12 @@ resources:
|
||||
- nextcloud-db-backup-credentials-prod.sops.yaml
|
||||
- nextcloud-db-backup-credentials.sops.yaml
|
||||
- nextcloud-db.yaml
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-bootstrap-secret.sops.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
- qbittorrent.yaml
|
||||
- semaphore-db-app.sops.yaml
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
|
||||
# =================================================================================================
|
||||
# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
|
||||
# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
|
||||
# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
|
||||
# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
|
||||
# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
|
||||
# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
|
||||
# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
|
||||
# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
|
||||
# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
|
||||
# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
|
||||
# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
|
||||
# schema has no word for.
|
||||
#
|
||||
# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
|
||||
# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
|
||||
# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
|
||||
# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
|
||||
# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
|
||||
# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
|
||||
#
|
||||
# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
|
||||
# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
|
||||
# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
|
||||
# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
|
||||
# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
|
||||
# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
|
||||
# can never be allowed to move.
|
||||
#
|
||||
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
|
||||
# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
|
||||
# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
|
||||
# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
|
||||
# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
|
||||
# and confusingly.
|
||||
#
|
||||
# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
|
||||
# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
|
||||
# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
|
||||
# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
|
||||
# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
|
||||
# filter below exists to refuse. Internal name, internal answer, no public record — which is also
|
||||
# why this change touches nothing under platform/externaldns/.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
|
||||
# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
|
||||
# not assumed (Cilium 1.20.1, prod, 2026-09-14):
|
||||
#
|
||||
# * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
|
||||
# CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
|
||||
# NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
|
||||
# landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
|
||||
# `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
|
||||
# * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
|
||||
# packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
|
||||
# nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
|
||||
# * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
|
||||
# rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
|
||||
# minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
|
||||
# re-election — and that re-election is random, so it may pick a backend-less node again.
|
||||
#
|
||||
# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
|
||||
# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
|
||||
# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
|
||||
# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
|
||||
# answering.
|
||||
#
|
||||
# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
|
||||
# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
|
||||
# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
|
||||
# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
|
||||
# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
|
||||
# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
|
||||
# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
|
||||
# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
|
||||
# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
|
||||
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
|
||||
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
|
||||
# range to a /24 that excludes the test client, after which the same query that had been answering
|
||||
# stopped being answered at all, and restoring it, after which it answered again.
|
||||
#
|
||||
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
|
||||
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
|
||||
# arrives from the ingress node's internal router address and Cilium resolves that source to the
|
||||
# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
|
||||
# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
|
||||
# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: pihole-dns-udp
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "pihole-dns"
|
||||
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
- { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: pihole-dns-tcp
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "pihole-dns" # same key + same IP => shares the VIP with -udp
|
||||
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
- { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
|
||||
---
|
||||
# =================================================================================================
|
||||
# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
|
||||
# =================================================================================================
|
||||
# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
|
||||
# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
|
||||
# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
|
||||
# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
|
||||
# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
|
||||
# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
|
||||
# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
|
||||
# it and no policy for it.
|
||||
#
|
||||
# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
|
||||
# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
|
||||
# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
|
||||
# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
|
||||
# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
|
||||
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
|
||||
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
|
||||
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
|
||||
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
|
||||
#
|
||||
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
|
||||
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
|
||||
# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
|
||||
# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
|
||||
#
|
||||
# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
|
||||
# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
|
||||
# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
|
||||
# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
|
||||
# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
|
||||
#
|
||||
# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
|
||||
# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
|
||||
# node-to-node forwarding that IS the Cluster-policy datapath both need them.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: pihole-allow-lan-dns-ingress
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- host # this node's kubelet probes
|
||||
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "53", protocol: UDP }
|
||||
- { port: "53", protocol: TCP }
|
||||
+12
-22
@@ -81,6 +81,15 @@ spec:
|
||||
image: pihole/pihole
|
||||
ports:
|
||||
- containerPort: 80
|
||||
env:
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
value: ALL
|
||||
- name: FTLCONF_dns_upstreams
|
||||
value: 8.8.8.8;8.8.4.4
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
@@ -223,6 +232,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -239,26 +249,6 @@ spec:
|
||||
- name: pihole
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: pihole
|
||||
namespace: pihole
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: pihole
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pihole
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- pihole
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -273,11 +263,11 @@ spec:
|
||||
- name: pihole.slis
|
||||
rules:
|
||||
- alert: pihole-down
|
||||
expr: up{job="pihole"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: pihole
|
||||
annotations:
|
||||
summary: pihole is down
|
||||
description: 'pihole has been unreachable for 5m (SLI: up).'
|
||||
description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -0,0 +1,77 @@
|
||||
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
|
||||
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
|
||||
#
|
||||
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
|
||||
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
|
||||
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
|
||||
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||
# not prose, so the upstream conditions below are described in words.
|
||||
#
|
||||
# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
|
||||
# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
|
||||
# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
|
||||
# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
|
||||
# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
|
||||
#
|
||||
# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
|
||||
# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
|
||||
# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
|
||||
# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
|
||||
#
|
||||
# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
|
||||
# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
|
||||
# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
|
||||
#
|
||||
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||
# the same file extracted from the running `plexinc/pms-docker:latest` image):
|
||||
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
|
||||
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
|
||||
# README: "If server is already logged in, this parameter is ignored."
|
||||
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
|
||||
# empty body. curl's exit status is never checked, the sed for the authentication-token element
|
||||
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
|
||||
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
|
||||
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
|
||||
# It does NOT crashloop.
|
||||
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
|
||||
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
|
||||
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
|
||||
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
|
||||
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
|
||||
# by hand at claim time.
|
||||
#
|
||||
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
|
||||
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||
# Never decrypt this file into a shell, a log, or a shared terminal.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plex-claim
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
type: Opaque
|
||||
stringData:
|
||||
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
|
||||
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
|
||||
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
|
||||
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-29T06:31:16Z"
|
||||
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+7
-22
@@ -132,6 +132,11 @@ spec:
|
||||
value: Etc/UTC
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
@@ -244,26 +249,6 @@ spec:
|
||||
- name: plex
|
||||
port: 32400
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: plex
|
||||
namespace: plex
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: plex
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- plex
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -278,11 +263,11 @@ spec:
|
||||
- name: plex.slis
|
||||
rules:
|
||||
- alert: plex-down
|
||||
expr: up{job="plex"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: plex
|
||||
annotations:
|
||||
summary: plex is down
|
||||
description: 'plex has been unreachable for 5m (SLI: up).'
|
||||
description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -0,0 +1,60 @@
|
||||
# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
|
||||
#
|
||||
# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
|
||||
# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
|
||||
# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
|
||||
# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
||||
# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
|
||||
# saved download-client credentials stop working at the next qbittorrent pod restart, silently
|
||||
# breaking Completed Download Handling; (b) a human operator has no stable login at all and must
|
||||
# read the pod log after every restart.
|
||||
#
|
||||
# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
|
||||
# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
|
||||
# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
|
||||
# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
|
||||
# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
|
||||
# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
|
||||
# credential, because the hook re-asserts it on every start rather than once.
|
||||
#
|
||||
# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
|
||||
# the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
|
||||
# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
|
||||
# entries store in their own databases (app state on their config PVCs, NOT in git — see
|
||||
# DECISIONS-LOG), so rotating it here means re-pointing both arrs.
|
||||
# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
|
||||
# payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
|
||||
#
|
||||
# NEVER put a value in a comment: SOPS does NOT encrypt comments.
|
||||
# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: qbittorrent-bootstrap
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
type: Opaque
|
||||
stringData:
|
||||
webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
|
||||
webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
|
||||
TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
|
||||
UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
|
||||
VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
|
||||
FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-13T05:23:23Z"
|
||||
mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -130,12 +130,31 @@ spec:
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
- name: QBT_WEBUI_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-username
|
||||
- name: QBT_WEBUI_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-password
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: downloads
|
||||
mountPath: /downloads
|
||||
subPath: downloads
|
||||
lifecycle:
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
|
||||
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
|
||||
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
@@ -325,6 +344,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -341,26 +361,6 @@ spec:
|
||||
- name: qbittorrent
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: qbittorrent
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- qbittorrent
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -375,11 +375,11 @@ spec:
|
||||
- name: qbittorrent.slis
|
||||
rules:
|
||||
- alert: qbittorrent-down
|
||||
expr: up{job="qbittorrent"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: qbittorrent
|
||||
annotations:
|
||||
summary: qbittorrent is down
|
||||
description: 'qbittorrent has been unreachable for 5m (SLI: up).'
|
||||
description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -0,0 +1,267 @@
|
||||
# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
|
||||
# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
|
||||
# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
|
||||
# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
|
||||
# tests run against the exact expressions Flux applies, never a copy.
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
|
||||
# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
|
||||
# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
|
||||
# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
|
||||
# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
|
||||
# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
|
||||
# workloads (verified on prod), and every rule below is pinned in BOTH directions:
|
||||
# * QUIET on the healthy state -> it is not permanently firing;
|
||||
# * FIRING when the workload has no ready replica -> it is not inert;
|
||||
# * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
|
||||
# (a rule that goes quiet because its series vanished) is itself alerted on.
|
||||
#
|
||||
# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
|
||||
# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
|
||||
# This is the "not permanently firing" half. It is also the test that catches a mis-spelled
|
||||
# label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
|
||||
# matches everything, so this test goes red instead of the rule going quietly wrong.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: jackett-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: pihole-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: qbittorrent-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
|
||||
# The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
|
||||
# would light up every app at once and be useless for routing.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 plex has no available replica - only plex-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: plex
|
||||
namespace: plex
|
||||
deployment: plex
|
||||
exp_annotations:
|
||||
summary: "plex is down"
|
||||
description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: jackett-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: pihole-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: qbittorrent-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts: []
|
||||
# Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
|
||||
# a single scrape blip does not page anyone.
|
||||
promql_expr_test:
|
||||
- expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
|
||||
eval_time: 2m
|
||||
exp_samples:
|
||||
- labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
|
||||
value: 0
|
||||
|
||||
- interval: 1m
|
||||
name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: gitea
|
||||
namespace: gitea
|
||||
deployment: gitea
|
||||
exp_annotations:
|
||||
summary: "gitea is down"
|
||||
description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: jackett-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: jackett
|
||||
namespace: jackett
|
||||
deployment: jackett
|
||||
exp_annotations:
|
||||
summary: "jackett is down"
|
||||
description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: pihole-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: pihole
|
||||
namespace: pihole
|
||||
deployment: pihole
|
||||
exp_annotations:
|
||||
summary: "pihole is down"
|
||||
description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: qbittorrent-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: qbittorrent
|
||||
namespace: qbittorrent
|
||||
deployment: qbittorrent
|
||||
exp_annotations:
|
||||
summary: "qbittorrent is down"
|
||||
description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
|
||||
# The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
|
||||
# the Deployment series for plex is simply not in the input at all, and plex-down must still
|
||||
# fire (via the `absent()` half), with the labels the selector's equality matchers carry.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: plex
|
||||
namespace: plex
|
||||
deployment: plex
|
||||
exp_annotations:
|
||||
summary: "plex is down"
|
||||
description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
|
||||
# series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
|
||||
# some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
|
||||
# is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
|
||||
# (a capital-R matcher selects nothing and would be just as inert).
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: vm-demo
|
||||
namespace: vm-demo
|
||||
name: vm-demo
|
||||
phase: running
|
||||
exp_annotations:
|
||||
summary: "vm-demo is down"
|
||||
description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
|
||||
|
||||
- interval: 1m
|
||||
name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
|
||||
input_series:
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: vm-demo
|
||||
namespace: vm-demo
|
||||
name: vm-demo
|
||||
phase: running
|
||||
exp_annotations:
|
||||
summary: "vm-demo is down"
|
||||
description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
|
||||
@@ -115,26 +115,6 @@ spec:
|
||||
- toEntities:
|
||||
- world
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: vm-demo
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- vm-demo
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -149,11 +129,11 @@ spec:
|
||||
- name: vm-demo.slis
|
||||
rules:
|
||||
- alert: vm-demo-down
|
||||
expr: up{job="vm-demo"} == 0
|
||||
expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: vm-demo
|
||||
annotations:
|
||||
summary: vm-demo is down
|
||||
description: 'vm-demo has been unreachable for 5m (SLI: up).'
|
||||
description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'
|
||||
@@ -4,7 +4,13 @@
|
||||
# against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
|
||||
# decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
|
||||
# in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
|
||||
# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
|
||||
# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
|
||||
# Gitea API and writes it here via `sops set`, without the value ever being printed.
|
||||
# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
|
||||
# it still decrypts, redirect: `sops -d <file> > /dev/null`.
|
||||
# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
|
||||
# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
|
||||
# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
@@ -14,7 +20,7 @@ metadata:
|
||||
app.kubernetes.io/name: act-runner
|
||||
type: Opaque
|
||||
stringData:
|
||||
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
|
||||
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -30,8 +36,8 @@ sops:
|
||||
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
|
||||
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-26T04:48:14Z"
|
||||
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
|
||||
lastmodified: "2026-09-29T05:48:35Z"
|
||||
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+69
-106
@@ -99,70 +99,79 @@ spec:
|
||||
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
||||
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
||||
automountServiceAccountToken: false
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
|
||||
#
|
||||
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
|
||||
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
|
||||
# running alongside them, and is terminated after them. kubelet holds `runner` until
|
||||
# this container's startupProbe passes, so the ordering is enforced by the kubelet
|
||||
# instead of hand-rolled shell in the runner.
|
||||
#
|
||||
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
|
||||
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
|
||||
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
|
||||
# depended on bare dollar-names surviving it untouched. There is now no shell here at
|
||||
# all, so there is nothing to escape or mis-escape.
|
||||
#
|
||||
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
|
||||
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
|
||||
# and only THEN begins listening, so the port can accept a connection while the certs
|
||||
# are absent or incomplete -- and act_runner hard-exits on a missing
|
||||
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
|
||||
# are usable and that the daemon answers.
|
||||
#
|
||||
# CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
|
||||
# introduced this claimed the conversion would REMOVE the one-per-start dockerd line
|
||||
# `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
|
||||
# appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
|
||||
# attempt, landing microseconds after dockerd begins listening and cut off by
|
||||
# timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
|
||||
# was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
|
||||
# startup line would slow every subsequent poll. Left documented rather than silenced.
|
||||
#
|
||||
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
|
||||
initContainers:
|
||||
- name: dind
|
||||
restartPolicy: Always # <- this field is what makes it a SIDECAR
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
- docker
|
||||
- --host=tcp://127.0.0.1:2376
|
||||
- --tlsverify
|
||||
- --tlscacert=/certs/client/ca.pem
|
||||
- --tlscert=/certs/client/cert.pem
|
||||
- --tlskey=/certs/client/key.pem
|
||||
- info
|
||||
periodSeconds: 1
|
||||
failureThreshold: 60
|
||||
containers:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
|
||||
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
|
||||
# that used to live here are gone: kubelet now holds this container until the dind
|
||||
# sidecar's startupProbe passes (see initContainers above).
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
@@ -198,52 +207,6 @@ spec:
|
||||
memory: 128Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
privileged: true
|
||||
env:
|
||||
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
|
||||
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
|
||||
- name: DOCKER_TLS_CERTDIR
|
||||
value: /certs
|
||||
volumeMounts:
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
|
||||
@@ -184,26 +184,6 @@ spec:
|
||||
- name: keycloak
|
||||
port: 8080
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- keycloak
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -218,11 +198,11 @@ spec:
|
||||
- name: keycloak.slis
|
||||
rules:
|
||||
- alert: keycloak-down
|
||||
expr: up{job="keycloak"} == 0
|
||||
expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: keycloak
|
||||
annotations:
|
||||
summary: keycloak is down
|
||||
description: 'keycloak has been unreachable for 5m (SLI: up).'
|
||||
description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
|
||||
@@ -290,26 +290,6 @@ spec:
|
||||
- name: nextcloud
|
||||
port: 80
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: nextcloud
|
||||
namespace: nextcloud
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: nextcloud
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: nextcloud
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- nextcloud
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -324,11 +304,11 @@ spec:
|
||||
- name: nextcloud.slis
|
||||
rules:
|
||||
- alert: nextcloud-down
|
||||
expr: up{job="nextcloud"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: nextcloud
|
||||
annotations:
|
||||
summary: nextcloud is down
|
||||
description: 'nextcloud has been unreachable for 5m (SLI: up).'
|
||||
description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
+3
-22
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -364,26 +365,6 @@ spec:
|
||||
- name: radarr
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: radarr
|
||||
namespace: radarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: radarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- radarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -398,11 +379,11 @@ spec:
|
||||
- name: radarr.slis
|
||||
rules:
|
||||
- alert: radarr-down
|
||||
expr: up{job="radarr"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: radarr
|
||||
annotations:
|
||||
summary: radarr is down
|
||||
description: 'radarr has been unreachable for 5m (SLI: up).'
|
||||
description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -227,6 +227,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -243,26 +244,6 @@ spec:
|
||||
- name: semaphore
|
||||
port: 3000
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: semaphore
|
||||
namespace: semaphore
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: semaphore
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: semaphore
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- semaphore
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -277,11 +258,11 @@ spec:
|
||||
- name: semaphore.slis
|
||||
rules:
|
||||
- alert: semaphore-down
|
||||
expr: up{job="semaphore"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: semaphore
|
||||
annotations:
|
||||
summary: semaphore is down
|
||||
description: 'semaphore has been unreachable for 5m (SLI: up).'
|
||||
description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
+3
-22
@@ -348,6 +348,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -364,26 +365,6 @@ spec:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: sonarr
|
||||
namespace: sonarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- sonarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -398,11 +379,11 @@ spec:
|
||||
- name: sonarr.slis
|
||||
rules:
|
||||
- alert: sonarr-down
|
||||
expr: up{job="sonarr"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
annotations:
|
||||
summary: sonarr is down
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -0,0 +1,245 @@
|
||||
# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
|
||||
# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
|
||||
# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
|
||||
# the flattened manifests are the RENDERED ones Flux applies.
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
|
||||
#
|
||||
# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
|
||||
# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
|
||||
# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
|
||||
# replica, and FIRING when the signal is ABSENT.
|
||||
#
|
||||
# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
|
||||
# Deployment this ServiceDefinition would otherwise imply — so its rule reads
|
||||
# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
|
||||
# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
|
||||
# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
|
||||
# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
|
||||
# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
|
||||
# = 2, and the only Deployment in that namespace is keycloak-operator.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
|
||||
input_series:
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
|
||||
values: '2+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
|
||||
values: '1+0x20'
|
||||
# The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
|
||||
# the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
|
||||
# namespace only (not the object) would be satisfied — or triggered — by these instead of by
|
||||
# the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
|
||||
values: '3+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
|
||||
# This is the anti-masking assertion: the neighbours in that namespace stay ready.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
|
||||
input_series:
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
|
||||
values: '3+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: keycloak
|
||||
namespace: keycloak
|
||||
statefulset: keycloak
|
||||
exp_annotations:
|
||||
summary: "keycloak is down"
|
||||
description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
|
||||
input_series:
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
|
||||
values: '2+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
|
||||
values: '0+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: nextcloud
|
||||
namespace: nextcloud
|
||||
deployment: nextcloud
|
||||
exp_annotations:
|
||||
summary: "nextcloud is down"
|
||||
description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: radarr
|
||||
namespace: radarr
|
||||
deployment: radarr
|
||||
exp_annotations:
|
||||
summary: "radarr is down"
|
||||
description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: semaphore
|
||||
namespace: semaphore
|
||||
deployment: semaphore
|
||||
exp_annotations:
|
||||
summary: "semaphore is down"
|
||||
description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
namespace: sonarr
|
||||
deployment: sonarr
|
||||
exp_annotations:
|
||||
summary: "sonarr is down"
|
||||
description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
|
||||
# only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
|
||||
# which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
|
||||
input_series:
|
||||
# One unrelated series so the test has an input at all; no wave-1 liveness series exists.
|
||||
- series: 'up{job="kube-state-metrics"}'
|
||||
values: '0+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: keycloak
|
||||
namespace: keycloak
|
||||
statefulset: keycloak
|
||||
exp_annotations:
|
||||
summary: "keycloak is down"
|
||||
description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: nextcloud
|
||||
namespace: nextcloud
|
||||
deployment: nextcloud
|
||||
exp_annotations:
|
||||
summary: "nextcloud is down"
|
||||
description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: radarr
|
||||
namespace: radarr
|
||||
deployment: radarr
|
||||
exp_annotations:
|
||||
summary: "radarr is down"
|
||||
description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: semaphore
|
||||
namespace: semaphore
|
||||
deployment: semaphore
|
||||
exp_annotations:
|
||||
summary: "semaphore is down"
|
||||
description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
namespace: sonarr
|
||||
deployment: sonarr
|
||||
exp_annotations:
|
||||
summary: "sonarr is down"
|
||||
description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
+2
-22
@@ -233,26 +233,6 @@ spec:
|
||||
- name: seerr
|
||||
port: 5055
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: seerr
|
||||
namespace: seerr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: seerr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: seerr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- seerr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -267,11 +247,11 @@ spec:
|
||||
- name: seerr.slis
|
||||
rules:
|
||||
- alert: seerr-down
|
||||
expr: up{job="seerr"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: seerr
|
||||
annotations:
|
||||
summary: seerr is down
|
||||
description: 'seerr has been unreachable for 5m (SLI: up).'
|
||||
description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
@@ -0,0 +1,69 @@
|
||||
# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
|
||||
# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
|
||||
# manifest is the RENDERED one Flux applies.
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
|
||||
#
|
||||
# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
|
||||
# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
|
||||
# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
|
||||
# signal-vanishes case.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
- interval: 1m
|
||||
name: wave-2 healthy - seerr has an available replica - quiet
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: seerr-down
|
||||
exp_alerts: []
|
||||
|
||||
- interval: 1m
|
||||
name: wave-2 seerr has no available replica - seerr-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
|
||||
values: '0+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: seerr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: seerr
|
||||
namespace: seerr
|
||||
deployment: seerr
|
||||
exp_annotations:
|
||||
summary: "seerr is down"
|
||||
description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
# Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
|
||||
# already be selecting — the distinction an inert rule cannot make.
|
||||
promql_expr_test:
|
||||
- expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
|
||||
eval_time: 2m
|
||||
exp_samples:
|
||||
- labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
|
||||
value: 0
|
||||
|
||||
- interval: 1m
|
||||
name: wave-2 seerr liveness series ABSENT - seerr-down still fires
|
||||
input_series:
|
||||
- series: 'up{job="kube-state-metrics"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: seerr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: seerr
|
||||
namespace: seerr
|
||||
deployment: seerr
|
||||
exp_annotations:
|
||||
summary: "seerr is down"
|
||||
description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
@@ -31,5 +31,15 @@ observability:
|
||||
metrics: true
|
||||
slis:
|
||||
- up # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
|
||||
# WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
|
||||
# `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
|
||||
# imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
|
||||
# is the catalog's declaration of intent for it. Without this override the rendered SLI would
|
||||
# read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
|
||||
# Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
|
||||
# statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
|
||||
livenessWorkload:
|
||||
kind: StatefulSet
|
||||
name: keycloak
|
||||
backup:
|
||||
policy: daily-offcluster # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)
|
||||
@@ -13,6 +13,93 @@ storage:
|
||||
size: 2Gi
|
||||
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
|
||||
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
|
||||
# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
|
||||
# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
|
||||
# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
|
||||
# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
|
||||
# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
|
||||
# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
|
||||
# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
|
||||
#
|
||||
# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
|
||||
# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
|
||||
# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
|
||||
# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
|
||||
# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
|
||||
# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
|
||||
# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
|
||||
# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
|
||||
# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
|
||||
# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
|
||||
# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
|
||||
# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
|
||||
#
|
||||
# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
|
||||
# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
|
||||
# configured through the UI is lost on the next reschedule — which is exactly what happened to the
|
||||
# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
|
||||
# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
|
||||
# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
|
||||
# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
|
||||
env:
|
||||
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
||||
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
||||
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
||||
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
||||
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
value: "ALL"
|
||||
# Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
|
||||
# router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
|
||||
# answers, and forwarding to a device that may forward back here invites a loop.
|
||||
- name: FTLCONF_dns_upstreams
|
||||
value: "8.8.8.8;8.8.4.4"
|
||||
# Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
|
||||
# because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
|
||||
# externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
|
||||
# announcement lease and the pihole pod land on different nodes, which is most of the time — and
|
||||
# Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
|
||||
# INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
|
||||
# apparent client's budget, and a normal evening of browsing from a handful of machines would trip
|
||||
# a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
|
||||
# off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
|
||||
# limit becomes meaningful again — and should be restored — if Cilium is ever switched to
|
||||
# `loadBalancer.mode: dsr`, which preserves the client address.
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: "0"
|
||||
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
||||
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
||||
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
||||
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
||||
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
||||
# changes nothing.
|
||||
#
|
||||
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
||||
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
||||
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
||||
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
||||
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
||||
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
||||
# is the annotation that means "this Service publishes its own hostname at its own address":
|
||||
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
||||
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
||||
# resolver address, are the complete set of exceptions today.
|
||||
#
|
||||
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
||||
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
||||
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
||||
#
|
||||
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
||||
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
||||
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
||||
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
|
||||
ingressRoutes:
|
||||
- host: pihole.${BASE_DOMAIN}
|
||||
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
||||
|
||||
+46
-9
@@ -32,11 +32,46 @@ env:
|
||||
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
||||
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
||||
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
||||
# single API POST) against the running server; the claim then persists on the /config PVC and
|
||||
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
||||
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
|
||||
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
|
||||
# workload). Plex cannot create libraries over the mounted bulk share while the server is
|
||||
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||
# half of that reasoning survived contact with the upstream source.
|
||||
#
|
||||
# A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
|
||||
# expires in MINUTES — read the countdown the claim page renders; do not budget against a number
|
||||
# written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
|
||||
# UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
|
||||
# for the whole procedure design — including a since-disproven claim that the delivery pipeline was
|
||||
# too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
|
||||
#
|
||||
# The live token is injected by the operator inside that window, with an explicit `flux reconcile`
|
||||
# to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
|
||||
# the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
|
||||
# committed value is the spent token from that claim rather than the original placeholder.
|
||||
#
|
||||
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||
# the running `:latest` image), NOT assumed:
|
||||
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
|
||||
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
|
||||
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
|
||||
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
|
||||
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
|
||||
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
|
||||
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
|
||||
# default 0, which continues startup even when a cont-init script fails.
|
||||
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
|
||||
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
|
||||
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
|
||||
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
|
||||
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
||||
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
||||
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
||||
@@ -56,10 +91,12 @@ storage:
|
||||
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
||||
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
||||
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
||||
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
||||
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
|
||||
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
|
||||
# exchange is then skipped on every later start. (The older note here claimed a
|
||||
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
|
||||
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
|
||||
# failure, and that is why one is committed.)
|
||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||
# downloads/ movies/ photos/ tv/).
|
||||
|
||||
@@ -44,6 +44,295 @@ env:
|
||||
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
|
||||
# readable to the EXISTING non-k3s consumers of the same dataset, and land
|
||||
# group-owned the same way the arrs' own writes do
|
||||
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
|
||||
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
|
||||
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
|
||||
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
||||
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
|
||||
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
|
||||
# tested valid against the then-current temporary password; both would have started failing at
|
||||
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
|
||||
# report a failed client, not a failed import).
|
||||
# * The human operator had no stable login: every restart meant reading the pod log.
|
||||
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
|
||||
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
|
||||
- name: QBT_WEBUI_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-username
|
||||
- name: QBT_WEBUI_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-password
|
||||
lifecycle:
|
||||
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
|
||||
#
|
||||
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
|
||||
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
|
||||
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
|
||||
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
|
||||
# the container start rather than a clock, because the failure this closes IS a container start:
|
||||
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
|
||||
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
|
||||
# restore or rebuild in which the app serves a random password and both arrs are broken. The
|
||||
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
|
||||
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
|
||||
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
|
||||
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
|
||||
# In-container, before the service starts, none of that exists.
|
||||
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
|
||||
# privilege than the CronJob alternative (R-SEC/least-privilege).
|
||||
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
|
||||
# keeps in its own state), and the schema documents this hook for exactly that use.
|
||||
#
|
||||
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
|
||||
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
|
||||
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
|
||||
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
|
||||
# credential — because the WebUI is not running yet. That assertion is the most valuable line
|
||||
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
|
||||
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
|
||||
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
|
||||
# mean a schema version bump plus a renderer change — a blast radius across every service's
|
||||
# golden — to buy something strictly weaker than what the hook already does.
|
||||
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
|
||||
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
|
||||
#
|
||||
# DURABILITY, the whole point — all three paths end with the SAME credential:
|
||||
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
|
||||
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
|
||||
# hook runs and re-asserts over whatever the snapshot contained.
|
||||
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
|
||||
# credential into it before the service is serving. Nothing outside git.
|
||||
#
|
||||
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
|
||||
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
|
||||
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
|
||||
# password is simply now known, fixed, and reproducible from git.
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
|
||||
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
|
||||
#
|
||||
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
|
||||
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
|
||||
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
|
||||
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
|
||||
# CLUSTER — but only there:
|
||||
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
|
||||
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
|
||||
# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
|
||||
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
|
||||
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
|
||||
# however it is run.
|
||||
set -u
|
||||
CONF=/config/qBittorrent/qBittorrent.conf
|
||||
SVC=/run/service/svc-qbittorrent
|
||||
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
|
||||
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
|
||||
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
|
||||
# env var for a lookup to read anyway.
|
||||
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
|
||||
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
|
||||
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
|
||||
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
|
||||
# logged "the WebUI rejected the durable
|
||||
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
|
||||
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
|
||||
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
|
||||
PORT=8080
|
||||
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
|
||||
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
|
||||
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
|
||||
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
|
||||
# credential from git was asserted and accepted on this start.
|
||||
log() {
|
||||
echo "[qbt-webui-cred] $*"
|
||||
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
|
||||
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
|
||||
}
|
||||
|
||||
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
|
||||
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
|
||||
# racing to create one it would then not overwrite.
|
||||
n=0
|
||||
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
|
||||
|
||||
waited=0
|
||||
attempt=0
|
||||
while [ "$attempt" -lt 3 ]; do
|
||||
attempt=$((attempt + 1))
|
||||
|
||||
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
|
||||
# 10 = rewritten, anything else = could not write.
|
||||
python3 - "$CONF" <<'PY'
|
||||
import base64, hashlib, hmac, os, stat, sys
|
||||
|
||||
conf = sys.argv[1]
|
||||
user = os.environ.get("QBT_WEBUI_USERNAME", "")
|
||||
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
|
||||
if not user or not pw:
|
||||
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
|
||||
raise SystemExit(1)
|
||||
|
||||
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
|
||||
# iterations, 16-byte random salt, 64-byte digest, stored as
|
||||
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
|
||||
# Get any one of them wrong and the app simply rejects every login.
|
||||
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
|
||||
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
|
||||
|
||||
def matches(raw):
|
||||
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
|
||||
v = raw.strip().strip('"')
|
||||
if v.startswith("@ByteArray(") and v.endswith(")"):
|
||||
v = v[len("@ByteArray("):-1]
|
||||
try:
|
||||
salt_b64, digest_b64 = v.split(":", 1)
|
||||
salt = base64.b64decode(salt_b64, validate=True)
|
||||
digest = base64.b64decode(digest_b64, validate=True)
|
||||
except Exception:
|
||||
return False
|
||||
if not salt or not digest:
|
||||
return False
|
||||
return hmac.compare_digest(
|
||||
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
|
||||
|
||||
try:
|
||||
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
|
||||
except FileNotFoundError:
|
||||
lines = []
|
||||
|
||||
def value_of(key):
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith(key + "="):
|
||||
return i, line.split("=", 1)[1]
|
||||
return -1, None
|
||||
|
||||
ui, uval = value_of(UKEY)
|
||||
pi, pval = value_of(PKEY)
|
||||
if uval == user and pval is not None and matches(pval):
|
||||
raise SystemExit(0) # converged - leave the file (and its salt) alone
|
||||
|
||||
salt = os.urandom(SALT_LEN)
|
||||
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
|
||||
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
|
||||
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
|
||||
|
||||
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
|
||||
if SECTION in out:
|
||||
at = out.index(SECTION) + 1
|
||||
else:
|
||||
if out and out[-1].strip():
|
||||
out.append("")
|
||||
out.append(SECTION)
|
||||
at = len(out)
|
||||
out[at:at] = desired
|
||||
|
||||
try:
|
||||
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
|
||||
tmp = conf + ".webui-cred.new"
|
||||
with open(tmp, "w", encoding="utf-8") as fh:
|
||||
fh.write("\n".join(out) + "\n")
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
|
||||
# cost the app the ability to save its own settings, so carry the ownership over.
|
||||
try:
|
||||
st = os.stat(conf)
|
||||
os.chown(tmp, st.st_uid, st.st_gid)
|
||||
os.chmod(tmp, stat.S_IMODE(st.st_mode))
|
||||
except FileNotFoundError:
|
||||
os.chown(tmp, int(os.environ.get("PUID", "1000")),
|
||||
int(os.environ.get("PGID", "1000")))
|
||||
os.chmod(tmp, 0o664)
|
||||
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
|
||||
except OSError as exc:
|
||||
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
|
||||
raise SystemExit(1)
|
||||
raise SystemExit(10)
|
||||
PY
|
||||
rc=$?
|
||||
|
||||
if [ "$rc" = 0 ]; then
|
||||
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
|
||||
elif [ "$rc" = 10 ]; then
|
||||
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
|
||||
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
|
||||
# have read the old, credential-less conf and be serving a random temporary password.
|
||||
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
|
||||
# Nothing to do when the service has not started yet - it will read the new file.
|
||||
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
|
||||
# 2026-09-13 failure spent the entire verification budget before the first login and
|
||||
# then fired attempts 2 and 3 with no wait left at all.
|
||||
sv=0
|
||||
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
|
||||
sv=$((sv + 1)); sleep 1
|
||||
done
|
||||
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
||||
log "qbittorrent-nox was already running - restarting it to load the credential"
|
||||
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
|
||||
sleep 3
|
||||
fi
|
||||
else
|
||||
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
|
||||
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
|
||||
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
|
||||
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
|
||||
#
|
||||
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
|
||||
# conflating them is what made this hook lie once already (see the PORT comment). An
|
||||
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
|
||||
# its remaining login attempts to produce two more identical wrong diagnoses.
|
||||
api_ok=0
|
||||
while [ "$waited" -lt 120 ]; do
|
||||
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
|
||||
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
|
||||
waited=$((waited + 1)); sleep 1
|
||||
done
|
||||
if [ "$api_ok" != 1 ]; then
|
||||
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
|
||||
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
|
||||
log "never submitted. Do not rotate the credential on the strength of this line." >&2
|
||||
exit 0
|
||||
fi
|
||||
body=$(curl -sS -m 10 \
|
||||
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
|
||||
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
|
||||
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
|
||||
if [ "$body" = "Ok." ]; then
|
||||
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
|
||||
exit 0
|
||||
fi
|
||||
log "WebUI rejected the durable credential (login said '$body')" >&2
|
||||
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
|
||||
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
|
||||
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
|
||||
# make the process RE-READ the file, so restart the supervised service and let the next
|
||||
# attempt re-verify. Without this the retry loop could never change its own outcome.
|
||||
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
||||
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
|
||||
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
|
||||
sleep 3
|
||||
fi
|
||||
done
|
||||
|
||||
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
|
||||
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
|
||||
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
|
||||
# app, and the arrs' own download-client health check surfaces a broken credential.
|
||||
exit 0
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
|
||||
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
|
||||
|
||||
+537
-17
@@ -80,7 +80,40 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
|
||||
BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
|
||||
GATEWAY_NAME = "shared"
|
||||
GATEWAY_NS = "gateway"
|
||||
# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
|
||||
# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
|
||||
# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
|
||||
# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
|
||||
# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
|
||||
# typo — "no DNS controller owns this route" — and because the one value that must never be
|
||||
# emitted here is external-dns's own, which a literal `none` cannot drift into.
|
||||
EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
EXTERNALDNS_CONTROLLER_DISOWN = "none"
|
||||
# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
|
||||
# assert the disown value is not it; never emitted.
|
||||
EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
|
||||
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
|
||||
MONITORING_NS = "monitoring" # where Prometheus runs; the only admitted scrape source
|
||||
|
||||
# The kube-state-metrics series a workload's readiness IS (r_prometheusrule). kube-state-metrics
|
||||
# is scraped by the bundled kube-prometheus-stack ServiceMonitor, i.e. these exist for EVERY
|
||||
# workload in the cluster with no per-service exporter — which is the whole point of using them
|
||||
# for `slis: [up]`. Each entry is (metric, label naming the object). VERIFIED on prod 2026-09-13:
|
||||
# kube_deployment_status_replicas_available returned a series for all ten deployed app
|
||||
# Deployments, and kube_statefulset_status_replicas_ready{namespace="keycloak",
|
||||
# statefulset="keycloak"} returned 2.
|
||||
LIVENESS_METRICS = {
|
||||
"Deployment": ("kube_deployment_status_replicas_available", "deployment"),
|
||||
"StatefulSet": ("kube_statefulset_status_replicas_ready", "statefulset"),
|
||||
"DaemonSet": ("kube_daemonset_status_number_available", "daemonset"),
|
||||
}
|
||||
# A KubeVirt VMI is not a replica count: kubevirt_vmi_info carries the phase as a LABEL, so
|
||||
# liveness is the EXISTENCE of a running-phase series. The value is lowercase `running` — a
|
||||
# capital-R matcher selects nothing and is just as inert (the same trap that made
|
||||
# kubevirt_vmi_phase_count{phase="Failed"} a no-op; verified on prod, where the one live VMI
|
||||
# reports phase="running").
|
||||
VMI_METRIC = "kubevirt_vmi_info"
|
||||
VMI_RUNNING_PHASE = "running"
|
||||
|
||||
# egress classes and auth modes fixed by docs/03 / docs/13 + the Kyverno enums.
|
||||
EGRESS_CLASSES = ("internet", "vpn")
|
||||
@@ -135,6 +168,26 @@ def apply_defaults(sd: dict) -> dict:
|
||||
obs = dict(s.get("observability") or {})
|
||||
obs.setdefault("metrics", True)
|
||||
obs.setdefault("slis", [])
|
||||
# metricsEndpoint is NEVER defaulted (schema 1.6.0). `metrics: true` says the platform
|
||||
# collects signals FOR this service; it cannot say the workload SERVES a Prometheus
|
||||
# endpoint, because for most of this catalog it does not (see r_servicemonitor). When the
|
||||
# endpoint IS declared, its name/path/interval carry defaults so the common case is
|
||||
# `metricsEndpoint: {port: 9100}`.
|
||||
ep = obs.get("metricsEndpoint")
|
||||
if ep:
|
||||
obs["metricsEndpoint"] = {
|
||||
**{"name": "metrics", "path": "/metrics", "interval": "30s"},
|
||||
**ep,
|
||||
}
|
||||
# livenessWorkload: WHICH Kubernetes object's readiness IS this service being up. Defaulted
|
||||
# from `kind` — a container renders a Deployment named after the service, a vm renders a
|
||||
# VirtualMachine whose VMI carries the same name — and overridable for the case where the
|
||||
# object that actually runs on the cluster is not the one this SD renders (keycloak: the
|
||||
# Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
|
||||
lw = dict(obs.get("livenessWorkload") or {})
|
||||
lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
|
||||
lw.setdefault("name", s["name"])
|
||||
obs["livenessWorkload"] = lw
|
||||
s["observability"] = obs
|
||||
|
||||
# storage entry class default is longhorn (replicated tier); readOnly default is False
|
||||
@@ -488,7 +541,12 @@ def r_deployment(s: dict) -> dict:
|
||||
|
||||
def r_service(s: dict):
|
||||
ports = _service_ports(s)
|
||||
if not ports:
|
||||
# A declared metrics endpoint becomes a NAMED Service port, because that name is the only
|
||||
# thing a ServiceMonitor endpoint can select on. Appended BELOW rather than inside
|
||||
# _service_ports so it never leaks into r_gateway_ingress_policy's north-south allow list —
|
||||
# the metrics port is for the monitoring namespace, not for the Gateway.
|
||||
ep = s["observability"].get("metricsEndpoint")
|
||||
if not ports and not ep:
|
||||
return None
|
||||
# forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
|
||||
# with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
|
||||
@@ -498,6 +556,10 @@ def r_service(s: dict):
|
||||
ports = ports + [
|
||||
{"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
|
||||
]
|
||||
if ep:
|
||||
ports = ports + [
|
||||
{"name": ep["name"], "port": ep["port"], "targetPort": ep["port"]}
|
||||
]
|
||||
return {
|
||||
"apiVersion": "v1",
|
||||
"kind": "Service",
|
||||
@@ -786,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list:
|
||||
|
||||
|
||||
def _route_meta(s: dict, route: dict) -> dict:
|
||||
"""HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
|
||||
`exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
|
||||
public DNS record, and now cannot)."""
|
||||
meta = {
|
||||
"name": f"{s['name']}-{route['port']}",
|
||||
"namespace": s["name"],
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
}
|
||||
ann = _auth_annotations(s)
|
||||
ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
|
||||
if ann:
|
||||
meta["annotations"] = ann
|
||||
return meta
|
||||
@@ -989,8 +1054,92 @@ def _auth_annotations(s: dict) -> dict:
|
||||
return {}
|
||||
|
||||
|
||||
def _externaldns_route_annotations(s: dict) -> dict:
|
||||
"""ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
|
||||
|
||||
`internal` renders NO public DNS record. That sentence was in the contract from the start and
|
||||
nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
|
||||
`internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
|
||||
Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
|
||||
(jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
|
||||
subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
|
||||
of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
|
||||
|
||||
THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
|
||||
`registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
|
||||
from memory. external-dns v0.21.0 `source/annotations/processors.go`:
|
||||
|
||||
func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
|
||||
value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
|
||||
if ok && value != ControllerValue { ...; return true }
|
||||
return false
|
||||
}
|
||||
|
||||
with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
|
||||
`ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
|
||||
`Endpoints()`, BEFORE any hostname is resolved:
|
||||
|
||||
if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
||||
|
||||
so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
|
||||
endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
|
||||
so the same annotation scopes the `service` source too if a rendered Service ever grows an
|
||||
external-dns hostname.
|
||||
|
||||
WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
|
||||
with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
|
||||
service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
|
||||
`catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
|
||||
`exposure: internal` to `public` is the only edit that can publish a name, which is exactly
|
||||
the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
|
||||
conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
|
||||
every gate run, including that this function reads `exposure` rather than naming services.
|
||||
|
||||
WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
|
||||
v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
|
||||
source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
|
||||
what is labelled" posture is available and is STRONGER by default. It is not taken here
|
||||
because its blast radius is not this defect's: seven published names come from hand-authored
|
||||
platform/apps manifests that carry no `exposure` field and that this renderer never sees
|
||||
(auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
|
||||
opt-in filter DELETES all seven the moment it is set, and whether each should stay published
|
||||
is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
|
||||
radius of the reported defect. The filter remains the available hardening once those seven
|
||||
are adjudicated — see platform/externaldns/helmrelease.yaml.
|
||||
"""
|
||||
if s["exposure"] == "internal":
|
||||
return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
|
||||
return {}
|
||||
|
||||
|
||||
def r_servicemonitor(s: dict):
|
||||
if not s["observability"]["metrics"]:
|
||||
"""ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
|
||||
|
||||
THIS FUNCTION USED TO RENDER A MONITOR UNCONDITIONALLY FROM `metrics: true`, hardcoding
|
||||
`endpoints: [{port: metrics}]`. No Service this renderer emits had ever defined a port of
|
||||
that name, so all twelve monitors discovered ZERO targets: no `up{job="<app>"}` series
|
||||
existed anywhere, `up{job="<app>"} == 0` could not fire, and `PrometheusTargetsMissing`
|
||||
could not see the hole either, because a target that was never discovered has no `up` series
|
||||
to be 0. Verified on prod 2026-09-13: `count by (job) (up)` listed 24 jobs and not one of
|
||||
the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing.
|
||||
|
||||
THE ROOT CAUSE WAS NOT A TYPO — it was that `metrics: true` cannot state a fact the workload
|
||||
does not have. Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and
|
||||
Nextcloud serve no Prometheus endpoint at all without a separate exporter; Gitea can serve
|
||||
one but only once its app.ini enables it. So the port name was never the missing piece. A
|
||||
monitor is now rendered only from `observability.metricsEndpoint`, which names the port the
|
||||
workload actually serves — and that same declaration renders the Service port (r_service) and
|
||||
the monitoring-namespace ingress allow (r_monitoring_ingress_policy), so the three halves of
|
||||
a working scrape cannot be authored apart.
|
||||
|
||||
`metrics: true` still means something without it: logs ship to Loki via the node agent, and
|
||||
liveness comes from kube-state-metrics (r_prometheusrule) — neither needs a per-service CR.
|
||||
"""
|
||||
obs = s["observability"]
|
||||
if not obs["metrics"]:
|
||||
return None
|
||||
ep = obs.get("metricsEndpoint")
|
||||
if not ep:
|
||||
return None
|
||||
return {
|
||||
"apiVersion": "monitoring.coreos.com/v1",
|
||||
@@ -1003,7 +1152,108 @@ def r_servicemonitor(s: dict):
|
||||
"spec": {
|
||||
"selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
|
||||
"namespaceSelector": {"matchNames": [s["name"]]},
|
||||
"endpoints": [{"port": "metrics", "interval": "30s"}],
|
||||
"endpoints": [
|
||||
{"port": ep["name"], "path": ep["path"], "interval": ep["interval"]}
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def r_monitoring_ingress_policy(s: dict):
|
||||
"""Let the monitoring namespace reach a declared metrics port (R-NET-02, R-OBS-01).
|
||||
|
||||
renderer-contract.md has always said the default-deny baseline carries "a standing allow ...
|
||||
for observability scraping from the monitoring namespace" — and no renderer output ever did.
|
||||
Every app namespace ships `ingress: []`, so a scrape of an app metrics port would be DROPPED
|
||||
by policy even with the Service port and the monitor endpoint in agreement. That is the same
|
||||
trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
|
||||
for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
|
||||
|
||||
Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
|
||||
monitoring namespace: an allow for a port nothing serves would be a standing hole for no
|
||||
benefit.
|
||||
"""
|
||||
ep = s["observability"].get("metricsEndpoint")
|
||||
if not ep or not s["observability"]["metrics"]:
|
||||
return None
|
||||
return {
|
||||
"apiVersion": "cilium.io/v2",
|
||||
"kind": "CiliumNetworkPolicy",
|
||||
"metadata": {
|
||||
"name": f"{s['name']}-allow-monitoring-scrape",
|
||||
"namespace": s["name"],
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
},
|
||||
"spec": {
|
||||
"endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
|
||||
"ingress": [
|
||||
{
|
||||
"fromEndpoints": [
|
||||
{"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
|
||||
],
|
||||
"toPorts": [
|
||||
{"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
|
||||
],
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _up_sli_rule(s: dict) -> dict:
|
||||
"""`slis: [up]` -> a `<name>-down` alert against a signal THAT EXISTS on this cluster.
|
||||
|
||||
IT USED TO READ `up{job="<name>"} == 0`. That is the textbook expression and it was inert
|
||||
for all twelve services in the catalog, because `up` only exists for a target Prometheus
|
||||
actually DISCOVERED and the rendered ServiceMonitor discovered none (see r_servicemonitor).
|
||||
Worse than a wrong threshold: an inert rule and a correct rule are indistinguishable on a
|
||||
healthy cluster, and the generic safety net cannot see the difference either — a target that
|
||||
was never discovered has no `up` series for PrometheusTargetsMissing to find at 0.
|
||||
|
||||
Rewriting the SLI against the app's own /metrics was NOT an option for most of this catalog:
|
||||
Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export no
|
||||
Prometheus metrics without a separate exporter sidecar per app. So the `up` SLI is expressed
|
||||
against kube-state-metrics, which IS scraped and DOES carry a series for every workload — the
|
||||
same idiom KeycloakDown in platform/monitoring/alerts.yaml has used since it was fixed.
|
||||
|
||||
The `or absent(...)` half is deliberate and load-bearing: it makes the DISAPPEARANCE of the
|
||||
signal fire the alert instead of silencing it. A rule that can only fire on `== 0` is a rule
|
||||
that goes quiet when the workload object (or kube-state-metrics) vanishes, which is precisely
|
||||
how the old expression failed. It is also why this rule cannot be permanently firing where the
|
||||
workload exists: the series is present and non-zero on a healthy cluster (verified on prod).
|
||||
"""
|
||||
lw = s["observability"]["livenessWorkload"]
|
||||
kind, obj = lw["kind"], lw["name"]
|
||||
ns = s["name"]
|
||||
if kind == "VirtualMachineInstance":
|
||||
sel = f'{VMI_METRIC}{{namespace="{ns}", name="{obj}", phase="{VMI_RUNNING_PHASE}"}}'
|
||||
expr = f"absent({sel})"
|
||||
desc = (
|
||||
f"{s['name']}: no KubeVirt VMI in phase {VMI_RUNNING_PHASE} for 5m "
|
||||
f"(SLI: up, via {VMI_METRIC})."
|
||||
)
|
||||
else:
|
||||
try:
|
||||
metric, label = LIVENESS_METRICS[kind]
|
||||
except KeyError:
|
||||
raise ValueError(
|
||||
f"{s['name']}: observability.livenessWorkload.kind {kind!r} has no liveness "
|
||||
f"metric; known kinds: {sorted(LIVENESS_METRICS)} + VirtualMachineInstance"
|
||||
) from None
|
||||
sel = f'{metric}{{namespace="{ns}", {label}="{obj}"}}'
|
||||
expr = f"{sel} == 0 or absent({sel})"
|
||||
desc = (
|
||||
f"{s['name']}: its {kind} has no available/ready replica for 5m, or the signal is "
|
||||
f"gone entirely (SLI: up, via {metric})."
|
||||
)
|
||||
return {
|
||||
"alert": f"{s['name']}-down".replace("_", "-"),
|
||||
"expr": expr,
|
||||
"for": "5m",
|
||||
"labels": {"severity": "critical", "service": s["name"]},
|
||||
"annotations": {
|
||||
"summary": f"{s['name']} is down",
|
||||
"description": desc,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -1016,18 +1266,7 @@ def r_prometheusrule(s: dict):
|
||||
rules = []
|
||||
for sli in slis:
|
||||
if sli == "up":
|
||||
rules.append(
|
||||
{
|
||||
"alert": f"{s['name']}-down".replace("_", "-"),
|
||||
"expr": f'up{{job="{s["name"]}"}} == 0',
|
||||
"for": "5m",
|
||||
"labels": {"severity": "critical", "service": s["name"]},
|
||||
"annotations": {
|
||||
"summary": f"{s['name']} is down",
|
||||
"description": f"{s['name']} has been unreachable for 5m (SLI: up).",
|
||||
},
|
||||
}
|
||||
)
|
||||
rules.append(_up_sli_rule(s))
|
||||
else:
|
||||
rules.append(
|
||||
{
|
||||
@@ -1386,6 +1625,16 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
|
||||
)
|
||||
)
|
||||
|
||||
mon_allow = r_monitoring_ingress_policy(s)
|
||||
if mon_allow:
|
||||
docs.append(
|
||||
(
|
||||
"CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): "
|
||||
f"{mon_allow['metadata']['name']}",
|
||||
mon_allow,
|
||||
)
|
||||
)
|
||||
|
||||
ie = r_internet_egress(s)
|
||||
docs.append(
|
||||
(
|
||||
@@ -1503,6 +1752,25 @@ EXTRA_UNITS: dict = {
|
||||
"gitea-actions-runner-secret.sops.yaml",
|
||||
],
|
||||
},
|
||||
# PiHole's DNS exposure: the udp/53 + tcp/53 LoadBalancer Services on a pinned VIP, plus the
|
||||
# narrow LAN ingress CiliumNetworkPolicy that admits them (apps/wave-0/pihole-dns.yaml).
|
||||
# `above: False` co-locates it in pihole's OWN wave: these are the pihole pod's own Services and
|
||||
# its own policy, applied in the same wave as the Deployment they select, exactly as the
|
||||
# rendered Service and CNPs in pihole.yaml are. It is NOT a runtime prerequisite of anything, so
|
||||
# there is nothing for a higher wave to gate on.
|
||||
#
|
||||
# It is a non-catalog unit because the ServiceDefinition schema cannot express a LoadBalancer
|
||||
# Service, a pinned lbipam VIP, a sharing key or UDP exposure at all (additionalProperties is
|
||||
# false, and `ingressRoutes` only renders HTTPRoutes on the L7 Gateway) — the same reason
|
||||
# apps/vms/unifi/50-service-l4.yaml and platform/crafty/service-l4.yaml are hand-written. The
|
||||
# split-horizon ANSWERS are in the catalog, as FTLCONF env on catalog/pihole.yaml.
|
||||
"pihole-dns": {
|
||||
"anchor": "pihole",
|
||||
"above": False,
|
||||
"files": [
|
||||
"pihole-dns.yaml",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -1576,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
convention:
|
||||
- `<svc>.yaml` -> wave(svc)
|
||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
|
||||
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
|
||||
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
|
||||
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
|
||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||
@@ -1602,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
w = waves[name]
|
||||
add(w, f"{name}.yaml")
|
||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
|
||||
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
|
||||
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
|
||||
# Placed only when the file exists, so services without one are unaffected.
|
||||
add(w, f"{name}-claim-secret.sops.yaml")
|
||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||
@@ -1902,6 +2179,96 @@ def _selftest() -> int:
|
||||
"a non-exposed service must not render a gateway-ingress allow (least privilege)",
|
||||
)
|
||||
|
||||
# 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
|
||||
# public DNS record and must be INCAPABLE of rendering one: its route carries
|
||||
# external-dns's own controller annotation set to something other than `dns-controller`,
|
||||
# which v0.21.0's gateway-httproute source (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch) uses to skip the route before any hostname is
|
||||
# resolved. `public` must NOT carry it, or the five public names would vanish instead.
|
||||
# Both directions are asserted here because only asserting the internal half would pass
|
||||
# just as happily if the renderer stamped EVERY route and silently unpublished the zone.
|
||||
dns_int_docs = render(
|
||||
{
|
||||
"name": "di",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(
|
||||
len(dns_int_routes) == 1,
|
||||
"an internal exposed service must still render its HTTPRoute (LAN reachability is "
|
||||
"unchanged — only the PUBLIC record is withdrawn)",
|
||||
)
|
||||
if dns_int_routes:
|
||||
iann = dns_int_routes[0]["metadata"].get("annotations", {})
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
|
||||
)
|
||||
check(
|
||||
iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
|
||||
"the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
|
||||
"the route rather than releasing it — the one value that inverts the guarantee)",
|
||||
)
|
||||
dns_pub_docs = render(
|
||||
{
|
||||
"name": "dp",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "public",
|
||||
"ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "native"},
|
||||
}
|
||||
)
|
||||
dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
|
||||
check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
|
||||
if dns_pub_routes:
|
||||
check(
|
||||
EXTERNALDNS_CONTROLLER_KEY
|
||||
not in dns_pub_routes[0]["metadata"].get("annotations", {}),
|
||||
"a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
|
||||
)
|
||||
# The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
|
||||
# ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
|
||||
# field is the likeliest way a new service arrives, and it is the case a per-service
|
||||
# allow-list would miss.
|
||||
dns_dflt_docs = render(
|
||||
{
|
||||
"name": "dd",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
|
||||
== EXTERNALDNS_CONTROLLER_DISOWN,
|
||||
"a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
|
||||
"unpublishable (default posture, renderer-contract)",
|
||||
)
|
||||
# The scoping annotation must COEXIST with the auth hints rather than replace them: both are
|
||||
# merged into one `annotations` map, and a forward-auth internal app is the overlap case.
|
||||
dns_fa_docs = render(
|
||||
{
|
||||
"name": "df",
|
||||
"source": {"image": "a:1"},
|
||||
"exposure": "internal",
|
||||
"ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
|
||||
}
|
||||
)
|
||||
dns_fa_ann = next(
|
||||
d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
|
||||
)["metadata"]["annotations"]
|
||||
check(
|
||||
dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
|
||||
and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
|
||||
"DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
|
||||
)
|
||||
|
||||
# 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
|
||||
# PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
|
||||
# declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
|
||||
@@ -2190,7 +2557,8 @@ def _selftest() -> int:
|
||||
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
|
||||
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
|
||||
present = {
|
||||
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"a.yaml", "a-claim-secret.sops.yaml",
|
||||
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
||||
"svc-db-backup-credentials-prod.sops.yaml",
|
||||
"gitea.yaml",
|
||||
@@ -2201,6 +2569,10 @@ def _selftest() -> int:
|
||||
plan = plan_apps_waves(plan_sds, present)
|
||||
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||
check(flat.get("a-claim-secret.sops.yaml") == 0,
|
||||
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
|
||||
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
|
||||
"CreateContainerConfigError")
|
||||
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
||||
@@ -2315,6 +2687,154 @@ def _selftest() -> int:
|
||||
"the escaped `$${...}` form must render (and reach the shell as `${...}`)",
|
||||
)
|
||||
|
||||
# 9) observability (R-OBS-01): `metrics: true` alone renders NO ServiceMonitor, and the `up`
|
||||
# SLI reads a signal that exists. This is the regression test for the defect that made all
|
||||
# twelve `<app>-down` rules structurally inert on prod (see r_servicemonitor).
|
||||
obs_sd = {
|
||||
"name": "obsdemo",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "obsdemo.${BASE_DOMAIN}", "port": 8080}],
|
||||
"auth": {"mode": "none"},
|
||||
"observability": {"metrics": True, "slis": ["up"]},
|
||||
}
|
||||
obs_docs = render(obs_sd)
|
||||
check(
|
||||
not [d for d in obs_docs if d[1]["kind"] == "ServiceMonitor"],
|
||||
"metrics: true with NO metricsEndpoint must render no ServiceMonitor — a monitor naming a "
|
||||
"port the Service does not define discovers zero targets and makes the SLI inert",
|
||||
)
|
||||
pr = next(d[1] for d in obs_docs if d[1]["kind"] == "PrometheusRule")
|
||||
up_rule = pr["spec"]["groups"][0]["rules"][0]
|
||||
check(up_rule["alert"] == "obsdemo-down", "slis: [up] must render <name>-down")
|
||||
check(
|
||||
"up{job=" not in up_rule["expr"],
|
||||
"the up SLI must NOT read up{job=...}: no repo-rendered scrape target produces that job, "
|
||||
"so the rule can never fire (verified inert on prod for all twelve apps)",
|
||||
)
|
||||
check(
|
||||
up_rule["expr"]
|
||||
== 'kube_deployment_status_replicas_available{namespace="obsdemo", deployment="obsdemo"} '
|
||||
'== 0 or absent(kube_deployment_status_replicas_available{namespace="obsdemo", '
|
||||
'deployment="obsdemo"})',
|
||||
"container up SLI must read kube_deployment_status_replicas_available with an absent() "
|
||||
f"half; got: {up_rule['expr']}",
|
||||
)
|
||||
|
||||
# 9a) livenessWorkload override: the object that RUNS is not always the one this SD renders.
|
||||
sts_docs = render(
|
||||
dict(
|
||||
obs_sd,
|
||||
name="obssts",
|
||||
observability={
|
||||
"metrics": True,
|
||||
"slis": ["up"],
|
||||
"livenessWorkload": {"kind": "StatefulSet", "name": "obssts"},
|
||||
},
|
||||
)
|
||||
)
|
||||
sts_expr = next(
|
||||
d[1] for d in sts_docs if d[1]["kind"] == "PrometheusRule"
|
||||
)["spec"]["groups"][0]["rules"][0]["expr"]
|
||||
check(
|
||||
"kube_statefulset_status_replicas_ready" in sts_expr,
|
||||
f"livenessWorkload kind StatefulSet must read the statefulset series; got: {sts_expr}",
|
||||
)
|
||||
|
||||
# 9b) vm kind: a VMI has no replica count, so liveness is a running-phase series EXISTING.
|
||||
vm_pr = next(
|
||||
d[1]
|
||||
for d in render(
|
||||
{
|
||||
"name": "obsvm",
|
||||
"kind": "vm",
|
||||
"source": {"vmTemplate": "quay.io/x/y:1"},
|
||||
"storage": [{"name": "root", "size": "6Gi"}],
|
||||
"observability": {"metrics": True, "slis": ["up"]},
|
||||
}
|
||||
)
|
||||
if d[1]["kind"] == "PrometheusRule"
|
||||
)
|
||||
check(
|
||||
vm_pr["spec"]["groups"][0]["rules"][0]["expr"]
|
||||
== 'absent(kubevirt_vmi_info{namespace="obsvm", name="obsvm", phase="running"})',
|
||||
"vm up SLI must read kubevirt_vmi_info with the LOWERCASE running phase: "
|
||||
f"{vm_pr['spec']['groups'][0]['rules'][0]['expr']}",
|
||||
)
|
||||
|
||||
# 9c) metricsEndpoint renders ALL THREE halves of a working scrape, and they AGREE. Any one of
|
||||
# them missing is an independent way for the scrape to silently discover/deliver nothing.
|
||||
ep_docs = render(
|
||||
dict(
|
||||
obs_sd,
|
||||
name="obsep",
|
||||
observability={
|
||||
"metrics": True,
|
||||
"slis": ["up"],
|
||||
"metricsEndpoint": {"port": 9100},
|
||||
},
|
||||
)
|
||||
)
|
||||
ep_svc = next(d[1] for d in ep_docs if d[1]["kind"] == "Service")
|
||||
ep_sm = next(d[1] for d in ep_docs if d[1]["kind"] == "ServiceMonitor")
|
||||
svc_port_names = {p["name"] for p in ep_svc["spec"]["ports"]}
|
||||
ep_ports = {e["port"] for e in ep_sm["spec"]["endpoints"]}
|
||||
check(
|
||||
ep_ports <= svc_port_names,
|
||||
f"ServiceMonitor endpoint port(s) {sorted(ep_ports)} must all exist on the Service "
|
||||
f"{sorted(svc_port_names)} — this exact disagreement is what made twelve monitors inert",
|
||||
)
|
||||
check(
|
||||
{"name": "metrics", "port": 9100, "targetPort": 9100} in ep_svc["spec"]["ports"],
|
||||
"metricsEndpoint must render the named Service port it declares",
|
||||
)
|
||||
mon_cnp = [
|
||||
d[1]
|
||||
for d in ep_docs
|
||||
if d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
and d[1]["metadata"]["name"] == "obsep-allow-monitoring-scrape"
|
||||
]
|
||||
check(
|
||||
len(mon_cnp) == 1,
|
||||
"a declared metricsEndpoint must render the monitoring-namespace ingress allow — the "
|
||||
"namespace default-deny has `ingress: []` and would DROP the scrape without it",
|
||||
)
|
||||
if mon_cnp:
|
||||
ing = mon_cnp[0]["spec"]["ingress"][0]
|
||||
check(
|
||||
ing["fromEndpoints"][0]["matchLabels"]["k8s:io.kubernetes.pod.namespace"]
|
||||
== MONITORING_NS
|
||||
and ing["toPorts"][0]["ports"] == [{"port": "9100", "protocol": "TCP"}],
|
||||
"the scrape allow must admit ONLY the monitoring namespace to ONLY the metrics port",
|
||||
)
|
||||
gw_cnp = next(
|
||||
d[1]
|
||||
for d in ep_docs
|
||||
if d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
and d[1]["metadata"]["name"] == "obsep-allow-gateway-ingress"
|
||||
)
|
||||
gw_ports = {p["port"] for p in gw_cnp["spec"]["ingress"][0]["toPorts"][0]["ports"]}
|
||||
check(
|
||||
"9100" not in gw_ports,
|
||||
"the metrics port must NOT be opened to the Gateway (north-south); it is for the "
|
||||
f"monitoring namespace only. Gateway allows: {sorted(gw_ports)}",
|
||||
)
|
||||
# A metricsEndpoint the renderer cannot express must be REFUSED, never rendered inert.
|
||||
try:
|
||||
render(
|
||||
dict(
|
||||
obs_sd,
|
||||
name="obsbad",
|
||||
observability={
|
||||
"metrics": True,
|
||||
"slis": ["up"],
|
||||
"livenessWorkload": {"kind": "CronJob", "name": "obsbad"},
|
||||
},
|
||||
)
|
||||
)
|
||||
check(False, "an unknown livenessWorkload.kind must raise, not render a bogus expression")
|
||||
except ValueError as exc:
|
||||
check("CronJob" in str(exc), "the refusal must name the unsupported kind")
|
||||
|
||||
if failures:
|
||||
for f in failures:
|
||||
sys.stderr.write(f"SELFTEST FAIL: {f}\n")
|
||||
|
||||
@@ -32,6 +32,12 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
|
||||
l1: ## L1 rendering checks (no cluster; live now)
|
||||
@"$(CI_DIR)l1.sh"
|
||||
|
||||
approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
|
||||
@./approval-policy.sh
|
||||
|
||||
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
||||
@./gitea-settings.sh
|
||||
|
||||
conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
|
||||
@"$(CI_DIR)conformance.sh"
|
||||
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# `ci/` — portable merge gate (runnable today, without Gitea)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
||||
|
||||
The same merge gate the Gitea Actions workflow runs
|
||||
|
||||
Executable
+117
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
|
||||
# what the pipeline verifies or the security posture it rests on.
|
||||
#
|
||||
# THE POLICY, and why it is shaped this way.
|
||||
#
|
||||
# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
|
||||
# pipeline is not good enough to be trusted for this, then it is not good enough to do its
|
||||
# intended job and needs to be improved. Manual approvers should only be for architecturally or
|
||||
# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
|
||||
#
|
||||
# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
|
||||
# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
|
||||
# A PR that weakens a check would be approved by the weakened check. The same applies to the
|
||||
# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
|
||||
# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
|
||||
# requires a human for them.
|
||||
#
|
||||
# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
|
||||
# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
|
||||
# policy cannot be widened without a human.
|
||||
#
|
||||
# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
|
||||
# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
|
||||
# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
|
||||
# BYPASS, not a gap. So: cannot determine -> BLOCK.
|
||||
#
|
||||
# EXIT CODES
|
||||
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
|
||||
# non-author APPROVED review is present
|
||||
# 1 sensitive paths touched without such a review, OR the policy could not be evaluated
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: approval policy (CI approves; humans approve changes to CI)"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
|
||||
[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool python3
|
||||
|
||||
# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
|
||||
# by branch protection, so this is genuinely not-applicable rather than a skipped check.
|
||||
EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
|
||||
PR="${PR_NUMBER:-}"
|
||||
if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
|
||||
ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
if [ -z "$TOKEN" ]; then
|
||||
err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
|
||||
err " BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
|
||||
summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
|
||||
exit 1
|
||||
fi
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
|
||||
PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
|
||||
REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
|
||||
for v in FILES_JSON PR_JSON REVIEWS_JSON; do
|
||||
jq -e . >/dev/null 2>&1 <<<"${!v}" || {
|
||||
err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
|
||||
summary "- approval-policy: **BLOCKED — Gitea unreadable**"
|
||||
exit 1; }
|
||||
done
|
||||
|
||||
AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
|
||||
[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
|
||||
|
||||
# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
|
||||
# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
|
||||
# never read. Caught before shipping, by running it.
|
||||
SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
|
||||
import json, os, fnmatch
|
||||
decl = json.load(open(os.environ["DECL"]))
|
||||
files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
|
||||
pats = decl["sensitive_paths"]
|
||||
hits = sorted({f for f in files for p in pats
|
||||
if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
|
||||
print("\n".join(hits))
|
||||
')"
|
||||
|
||||
if [ -z "$SENSITIVE" ]; then
|
||||
ok "no architecturally/security-relevant paths touched — the GATE is the approver."
|
||||
summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo " sensitive paths touched:" >&2
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
|
||||
APPROVER="$(jq -r --arg a "$AUTHOR" \
|
||||
'[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
|
||||
<<<"$REVIEWS_JSON")"
|
||||
|
||||
if [ -n "$APPROVER" ]; then
|
||||
ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
|
||||
summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
|
||||
err " review from anyone other than its author ('${AUTHOR}')."
|
||||
err " A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
|
||||
err " honestly approve for itself. Get a human review, or move the change out of:"
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
err " The list lives in infra/gitea/approval-policy.json, with a reason per entry."
|
||||
summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
|
||||
exit 1
|
||||
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
||||
#
|
||||
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
||||
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
||||
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
||||
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
||||
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
||||
#
|
||||
# MEASURED 2026-09-29, before this guard existed:
|
||||
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
||||
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
||||
# "[ ok ] L2/L3 conformance passed for touched subset"
|
||||
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
||||
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
||||
# it because nobody had ever set the variable.
|
||||
#
|
||||
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
||||
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
||||
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
||||
# makes the stage itself honour that instead of reporting a pass.
|
||||
JUNIT="${CONF_ART}/report.junit.xml"
|
||||
if [ -f "$JUNIT" ]; then
|
||||
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
||||
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
||||
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
||||
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
||||
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
||||
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
||||
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
||||
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
||||
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
||||
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
||||
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
||||
exit 1
|
||||
fi
|
||||
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
||||
fi
|
||||
|
||||
ok "L2/L3 conformance passed for touched subset"
|
||||
summary "- conformance-l2-l3: **passed**"
|
||||
+41
@@ -112,6 +112,8 @@ run_stage lint "${CI_DIR}/lint.sh"
|
||||
run_stage l0 "${CI_DIR}/l0.sh"
|
||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||
run_stage l1 "${CI_DIR}/l1.sh"
|
||||
run_stage approval-policy "${CI_DIR}/approval-policy.sh"
|
||||
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
||||
run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
|
||||
echo >&2
|
||||
@@ -188,6 +190,45 @@ case "${OUTCOME[spec-immutability]}" in
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
|
||||
# is the approver, per operator intent. This stage is the single exception: a change to the
|
||||
# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
|
||||
# cannot honestly be approved BY the thing it changes, so those paths still need a human.
|
||||
#
|
||||
# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
|
||||
# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
|
||||
# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
|
||||
case "${OUTCOME[approval-policy]}" in
|
||||
fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
|
||||
skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
|
||||
pass) ok "approval-policy: green" ;;
|
||||
esac
|
||||
|
||||
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
||||
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
||||
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
||||
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
|
||||
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
|
||||
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
|
||||
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
|
||||
# token is wired, and the skip becomes a block as it should.
|
||||
case "${OUTCOME[gitea-settings]}" in
|
||||
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
|
||||
skip)
|
||||
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
|
||||
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
|
||||
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
|
||||
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
|
||||
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
|
||||
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
|
||||
NOT_ESTABLISHED+=("gitea-settings")
|
||||
fi ;;
|
||||
pass) ok "gitea-settings: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
|
||||
Executable
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
|
||||
# match what git declares.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS AT ALL
|
||||
# ----------------------------
|
||||
# Every other check in this repo reads the repository, the rendered manifests, or the
|
||||
# cluster. NONE of them can see Gitea's own database — and that is where branch
|
||||
# protection and runner registration live. On 2026-09-29 that blind spot was measured:
|
||||
#
|
||||
# * `main` had NO branch protection, and
|
||||
# * ZERO Actions runners were registered,
|
||||
#
|
||||
# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
|
||||
# and human in the loop repeated that claim because the repository said so. PR #1 — the
|
||||
# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
|
||||
# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
|
||||
# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
|
||||
# merged with zero approvals recorded against `required_approvals: 1`.
|
||||
#
|
||||
# Three separate failures, one shape: a setting that READ as enforced while not being, with
|
||||
# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
|
||||
# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
|
||||
#
|
||||
# WHAT IT ASSERTS
|
||||
# 1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
|
||||
# LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
|
||||
# 2. at least `.runners_online_min` Actions runners report status "online".
|
||||
#
|
||||
# (2) is not decoration. A required status check with no runner is the deadlock this repo
|
||||
# nearly walked into: every PR pends forever, including the one that would fix it.
|
||||
#
|
||||
# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
|
||||
# `protected`, `enable_status_check` and `required_approvals` but NOT
|
||||
# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
|
||||
# only account that merges. This stage queries /branch_protections/ deliberately.
|
||||
#
|
||||
# EXIT CODES (the gate contract)
|
||||
# 0 live Gitea matches the declaration
|
||||
# 1 DIVERGENCE — something drifted, or was never applied. Blocks the merge.
|
||||
# 3 SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
|
||||
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
|
||||
# that an unwired credential cannot lock the repository.
|
||||
#
|
||||
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
|
||||
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
|
||||
# the drift it exists to catch is still undetected.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
log "Stage: Gitea platform settings match the declaration"
|
||||
|
||||
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
|
||||
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
|
||||
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
|
||||
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
|
||||
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
|
||||
# it looked like it had worked. A token was minted, no secret existed, and this stage would
|
||||
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
|
||||
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||
|
||||
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
|
||||
require_tool jq
|
||||
require_tool curl
|
||||
|
||||
if [ -z "$TOKEN" ]; then
|
||||
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
|
||||
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
|
||||
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
|
||||
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
|
||||
BRANCH="$(jq -r '.branch' "$DECL")"
|
||||
LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
|
||||
if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
|
||||
skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
|
||||
skip " lacks permission). Not treating this as a pass."
|
||||
summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
RC=0
|
||||
|
||||
# --- 1. every declared protection key must match the live rule, exactly ---------------
|
||||
while IFS= read -r key; do
|
||||
want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
|
||||
got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
|
||||
if [ "$want" = "$got" ]; then
|
||||
ok " ${key} = ${got}"
|
||||
else
|
||||
err " ${key}: DECLARED ${want} but LIVE is ${got}"
|
||||
RC=1
|
||||
fi
|
||||
done < <(jq -r '.protection | keys[]' "$DECL")
|
||||
|
||||
# --- 2. a runner must actually be online ----------------------------------------------
|
||||
MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
|
||||
RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
|
||||
N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
|
||||
if [ "${N:-0}" -ge "$MIN" ]; then
|
||||
ok " runners online: ${N} (>= ${MIN})"
|
||||
else
|
||||
err " runners online: ${N}, DECLARED minimum ${MIN}."
|
||||
err " A required status check with no runner leaves every PR pending forever —"
|
||||
err " including the PR that would fix it. This is the 2026-09-29 outage condition."
|
||||
RC=1
|
||||
fi
|
||||
|
||||
if [ "$RC" -eq 0 ]; then
|
||||
ok "Gitea settings match the declaration."
|
||||
summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
|
||||
else
|
||||
err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
|
||||
err " Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
|
||||
err " state is what you now want, change the DECLARATION in the same commit — deliberately."
|
||||
summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
|
||||
fi
|
||||
exit "$RC"
|
||||
@@ -79,12 +79,37 @@ else
|
||||
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
||||
find "$l0_render" -type f -name '*.yaml' -print0 \
|
||||
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
||||
if kyverno test "$l0_render"; then
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
|
||||
# error, bad match block, malformed JMESPath) does not fail the run: every result row
|
||||
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
|
||||
#
|
||||
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
|
||||
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
|
||||
# rows came back `Skip / Invalid Policy`, and the summary read:
|
||||
# Test Summary: 9 tests passed and 0 tests failed
|
||||
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
|
||||
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
|
||||
#
|
||||
# This is not specific to that policy: a typo in restrict-host-namespaces or
|
||||
# secret-provenance would silently disable it and the gate would stay green. Scan the
|
||||
# output for the load failure the exit code hides.
|
||||
l0_kout="$(mktemp)"
|
||||
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
|
||||
if grep -q 'Invalid Policy' "$l0_kout"; then
|
||||
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
|
||||
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
|
||||
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
|
||||
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
|
||||
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
|
||||
rc=1
|
||||
else
|
||||
ok "Kyverno L0 policy tests passed"
|
||||
summary "- kyverno-l0: **passed**"
|
||||
fi
|
||||
else
|
||||
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
||||
fi
|
||||
rm -f "$l0_kout"
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
|
||||
@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
# --include-scope: union the touched subset with every L1 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
|
||||
# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
|
||||
# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
|
||||
# "someone must remember" is the failure mode this whole change removes; see
|
||||
# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
|
||||
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
|
||||
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
|
||||
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
|
||||
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
|
||||
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
|
||||
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
l1_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
|
||||
@@ -136,6 +136,98 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
||||
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
||||
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
|
||||
#
|
||||
# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
|
||||
# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
|
||||
# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
|
||||
# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
|
||||
# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
|
||||
# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
|
||||
#
|
||||
# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
|
||||
# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
|
||||
#
|
||||
# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
|
||||
# from the PATH and not from the content scan.
|
||||
|
||||
# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
|
||||
# under test here, not just the presence of both arms.
|
||||
git -C "$TMP" checkout -q -b feature-apps-vms main
|
||||
mkdir -p "$TMP/apps/vms/demo"
|
||||
printf 'kind: VirtualMachine\nmetadata:\n name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
|
||||
OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
|
||||
assert_present "R-PLAT-03" "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
|
||||
assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
|
||||
assert_absent "R-PLAT-02" "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
|
||||
|
||||
# A plain ServiceDefinition under apps/ gets the container-workload set.
|
||||
git -C "$TMP" checkout -q -b feature-apps main
|
||||
mkdir -p "$TMP/apps/wave-1"
|
||||
printf 'kind: ServiceDefinition\nmetadata:\n name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
|
||||
OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
|
||||
assert_present "R-PLAT-02" "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
|
||||
assert_present "R-NET-01" "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
|
||||
assert_present "R-NET-06" "$OUT_APPS" "apps/* attributes the egress-class requirements"
|
||||
assert_present "R-IDP-02" "$OUT_APPS" "apps/* attributes the auth-mode requirements"
|
||||
|
||||
# homepage / headlamp ARE the command-centre surface.
|
||||
git -C "$TMP" checkout -q -b feature-c2 main
|
||||
mkdir -p "$TMP/platform/headlamp"
|
||||
printf 'kind: Deployment\nmetadata:\n name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
|
||||
OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
|
||||
assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
|
||||
assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
|
||||
|
||||
# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
|
||||
git -C "$TMP" checkout -q -b feature-assembly main
|
||||
mkdir -p "$TMP/platform"
|
||||
printf 'resources:\n - cilium\n - longhorn\n' > "$TMP/platform/kustomization.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
|
||||
OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
|
||||
assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
|
||||
#
|
||||
# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
|
||||
# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
|
||||
# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
|
||||
# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
|
||||
# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
|
||||
# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
|
||||
#
|
||||
# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
|
||||
# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
|
||||
# somehow matched only the digit form would still fail here.
|
||||
git -C "$TMP" checkout -q -b feature5 main
|
||||
cat > "$TMP/notes/c2.yaml" <<'YAML'
|
||||
kind: Test
|
||||
metadata:
|
||||
labels:
|
||||
conformance.home.k3s/R-C2-01: "true"
|
||||
conformance.home.k3s/R-C2-02: "true"
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
YAML
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
|
||||
OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
|
||||
assert_present "R-C2-01" "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
|
||||
assert_present "R-C2-02" "$OUT5" "the second such ID is attributed too (R-C2-02)"
|
||||
assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
|
||||
# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
|
||||
# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
|
||||
# noise (over-broad selection, never a false green, but noisy and misleading).
|
||||
assert_absent "R-1-01" "$OUT5" "a purely numeric workstream segment is not invented from thin air"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
||||
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -177,6 +177,23 @@ path_requirements() {
|
||||
case "$1" in
|
||||
catalog/*) echo "R-GITOPS-02" ;;
|
||||
schemas/*) echo "R-GITOPS-02" ;;
|
||||
# apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
|
||||
# manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
|
||||
# adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
|
||||
# checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
|
||||
# class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
|
||||
# violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
|
||||
# catch it at apply time" is post-merge, and the gate exists to catch it before.
|
||||
#
|
||||
# ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
|
||||
# apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
|
||||
# subject to default-deny (R-NET-01).
|
||||
apps/vms/*) echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
|
||||
# Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
|
||||
# workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
|
||||
# (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
|
||||
# default-deny (R-NET-01).
|
||||
apps/*) echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
|
||||
platform/kyverno/*) echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
|
||||
platform/cilium/*) echo "R-NET-01 R-NET-02 R-NET-05" ;;
|
||||
platform/longhorn/*) echo "R-STATE-01 R-STATE-02" ;;
|
||||
@@ -208,6 +225,39 @@ path_requirements() {
|
||||
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
# The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
|
||||
# rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
|
||||
# itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
|
||||
# docs/12-command-center-ui.md), so editing them must select the C2 tests.
|
||||
platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
|
||||
# KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
|
||||
platform/kubevirt/*) echo "R-PLAT-03 R-OBS-01" ;;
|
||||
# The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
|
||||
# TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
|
||||
platform/gateway/*) echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
|
||||
# CoreDNS is cluster DNS resolution (R-DNS-01).
|
||||
platform/coredns/*) echo "R-DNS-01" ;;
|
||||
# crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
|
||||
# its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
|
||||
platform/crafty/*) echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
|
||||
# Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
|
||||
platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
|
||||
# runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
|
||||
# resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
|
||||
# is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
|
||||
# runbook change is an observability change.
|
||||
runbooks/*) echo "R-OBS-04" ;;
|
||||
# platform-config/ is the env-specific configuration Flux substitutes into and applies.
|
||||
platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
|
||||
# platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
|
||||
# Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
|
||||
# file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
|
||||
# change. That is the most consequential single-line edit in the repo and it attributed nothing.
|
||||
platform/kustomization.yaml) echo "R-GITOPS-01" ;;
|
||||
# .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
|
||||
# here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
|
||||
.sops.yaml) echo "R-SEC-01 R-GITOPS-01" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
|
||||
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
|
||||
@@ -345,6 +395,21 @@ main() {
|
||||
done <<< "$files"
|
||||
|
||||
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
|
||||
#
|
||||
# THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
|
||||
# contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
|
||||
# in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
|
||||
# to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
|
||||
# Every other consumer in the repo already allows the digit: test_requirements_for_path()
|
||||
# twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
|
||||
# conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
|
||||
# dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
|
||||
# retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
|
||||
# path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
|
||||
# filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
|
||||
# reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
|
||||
# (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
|
||||
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
|
||||
# `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
|
||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||
@@ -358,7 +423,7 @@ main() {
|
||||
| grep -vE '^(\+\+\+|---)' \
|
||||
| sed -E 's/^[+-]//' \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| grep -hoE 'R-[A-Z]+-[0-9]+' \
|
||||
| grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
|
||||
| tr '\n' ' ' || true)"
|
||||
|
||||
# Normalize: split, unique, sort. Emit to stdout.
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# clusters/
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
||||
|
||||
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
||||
|
||||
@@ -99,7 +99,9 @@ data:
|
||||
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
|
||||
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
|
||||
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
|
||||
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
|
||||
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
||||
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
||||
#
|
||||
@@ -159,6 +161,30 @@ data:
|
||||
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
# PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
|
||||
# THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
|
||||
# terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
|
||||
# LAN clients are pointed at this address as their resolver, and it has to stay put across
|
||||
# reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
|
||||
#
|
||||
# WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
|
||||
# the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
|
||||
# answers where services live" with "a service", and it would put LAN resolution behind the
|
||||
# Gateway Service's own lifecycle. Keep them separate.
|
||||
#
|
||||
# dev -> 192.168.1.56 (base default; next free after crafty .55 in dev's .53-.60 pool)
|
||||
# prod -> 192.168.2.23 (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
|
||||
PIHOLE_DNS_VIP: "192.168.1.56" # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
|
||||
|
||||
# The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
|
||||
# peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
|
||||
# from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
|
||||
# CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
|
||||
# same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
|
||||
#
|
||||
# dev -> 192.168.1.0/24 (Proxmox LAN; nodes .50/.51/.52)
|
||||
# prod -> 192.168.2.0/24 (prod LAN; nodes .10/.11/.12)
|
||||
LAN_CIDR: "192.168.1.0/24" # ingress scope for the PiHole DNS listener
|
||||
|
||||
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
|
||||
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
|
||||
#
|
||||
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
|
||||
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
|
||||
# meant to verify is worse than no harness.
|
||||
#
|
||||
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
|
||||
# waves needs it.
|
||||
#
|
||||
# NOTE: this reconciles RBAC that is INERT until
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
|
||||
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
|
||||
# trigger identity can exist while the policy bounding it does not.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: conformance
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./conformance/incluster
|
||||
@@ -34,8 +34,12 @@ spec:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/homepage
|
||||
# Homepage references no ${VARS} today; the substitution is retained for parity with the other
|
||||
# tiers / future overlay knobs.
|
||||
# LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
|
||||
# references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
|
||||
# configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
|
||||
# the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
|
||||
# front door points at a host that does not exist — while Flux still reports this Kustomization
|
||||
# Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
|
||||
@@ -33,6 +33,7 @@ resources:
|
||||
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
||||
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
||||
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
||||
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
|
||||
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
||||
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
||||
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
||||
|
||||
@@ -122,6 +122,23 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
|
||||
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
|
||||
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
|
||||
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
|
||||
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
|
||||
# as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-vms
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
|
||||
@@ -113,6 +113,23 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
|
||||
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
|
||||
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
|
||||
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
|
||||
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
|
||||
# as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-vms
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
||||
#
|
||||
# ############################################################################################
|
||||
@@ -349,8 +366,16 @@ patches:
|
||||
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
|
||||
# size cap before 30d if series volume is high, keeping the disk from filling either way.
|
||||
- op: replace
|
||||
# LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
|
||||
# what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
|
||||
# not the head's in-flight segments, and not compaction scratch, and a large compaction can need
|
||||
# roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
|
||||
# /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
|
||||
# absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
|
||||
# at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
|
||||
# grow the PVC rather than the fraction of it.
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
||||
value: "45GiB"
|
||||
value: "36GiB"
|
||||
- op: replace
|
||||
path: /data/MONITORING_LOKI_PVC_SIZE
|
||||
value: "50Gi"
|
||||
@@ -440,6 +465,23 @@ patches:
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.2.22"
|
||||
# PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
|
||||
# 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
|
||||
# 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
|
||||
# IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
|
||||
# (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
|
||||
# broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
|
||||
# it either, and it is inside the pool block the operator already asserted sits outside the
|
||||
# SOHO router's DHCP scope.
|
||||
- op: replace
|
||||
path: /data/PIHOLE_DNS_VIP
|
||||
value: "192.168.2.23"
|
||||
# The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
|
||||
# NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
|
||||
# It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
|
||||
- op: replace
|
||||
path: /data/LAN_CIDR
|
||||
value: "192.168.2.0/24"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
|
||||
# ------------------------------------------------------------------------------------------
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
||||
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
||||
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# In-cluster conformance Job (L2/L3)
|
||||
|
||||
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
||||
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
||||
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
||||
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
||||
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
||||
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
||||
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
||||
> coverage hole, not a phase.
|
||||
>
|
||||
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
||||
> documentation rather than measurement, and `main` was unprotected with zero runners
|
||||
> registered the whole time. **Query the live API before asserting platform state.**
|
||||
|
||||
|
||||
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
||||
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
||||
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
|
||||
#
|
||||
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
|
||||
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
|
||||
# reconciling it would create a permanent Job object and fight the trigger for ownership.
|
||||
#
|
||||
# The identities here are inert without
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
|
||||
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
|
||||
# half — see the header of rbac.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- rbac.yaml
|
||||
@@ -174,3 +174,68 @@ subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-runner
|
||||
namespace: conformance
|
||||
---
|
||||
# =============================================================================
|
||||
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
|
||||
#
|
||||
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
|
||||
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
|
||||
# that can `create jobs` in this namespace can create a Job with
|
||||
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
|
||||
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
|
||||
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
|
||||
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
|
||||
# access") while appearing to satisfy it.
|
||||
#
|
||||
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
|
||||
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
|
||||
# the necessary half; that policy is the sufficient half. Neither works alone — do not
|
||||
# grant this Role without that policy reconciled and ENFORCING.
|
||||
#
|
||||
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
rules:
|
||||
# Create the run, and watch it to completion.
|
||||
- apiGroups: [batch]
|
||||
resources: [jobs]
|
||||
verbs: [create, get, list, watch]
|
||||
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
|
||||
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
|
||||
# them from the log, so no execution channel into the runner pod is ever granted.
|
||||
- apiGroups: [""]
|
||||
resources: [pods, pods/log]
|
||||
verbs: [get, list, watch]
|
||||
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
|
||||
# failed run), no secrets, no configmaps, no exec, no create on pods.
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: conformance-trigger
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-trigger
|
||||
namespace: conformance
|
||||
@@ -489,6 +489,36 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
|
||||
return "unknown"
|
||||
|
||||
|
||||
def context_env() -> str:
|
||||
"""The env a run TARGETED, derived from the live kube context — never guessed.
|
||||
|
||||
WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
|
||||
(`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
|
||||
recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
|
||||
harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
|
||||
the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
|
||||
a proof that lies about where it was proven is worse than no proof, because the ledger's
|
||||
whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
|
||||
|
||||
Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
|
||||
from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
|
||||
name: an unreadable context must not silently attribute a proof to an environment.
|
||||
"""
|
||||
try:
|
||||
p = subprocess.run(["kubectl", "config", "current-context"],
|
||||
capture_output=True, text=True, timeout=15)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return "unknown"
|
||||
if p.returncode != 0:
|
||||
return "unknown"
|
||||
ctx = (p.stdout or "").strip()
|
||||
if not ctx:
|
||||
return "unknown"
|
||||
# contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
|
||||
# unexpected context is visible in the ledger rather than being coerced into a known name.
|
||||
return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
|
||||
|
||||
|
||||
def server_kube_version() -> str:
|
||||
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
|
||||
for args in (["kubectl", "version", "-o", "json"],
|
||||
@@ -883,7 +913,9 @@ def main() -> int:
|
||||
help=f"append-only ledger (default {DEFAULT_LEDGER})")
|
||||
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
|
||||
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
|
||||
help="env the run targeted; derived from the kube context when "
|
||||
"unset (never defaulted to a cluster name)")
|
||||
ap.add_argument("--chainsaw-version", default=None,
|
||||
help="override the versions.env pin (recorded verbatim)")
|
||||
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
|
||||
|
||||
+2
-2
@@ -201,7 +201,7 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||||
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
||||
# to the real ledger; operators should never set it.
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
||||
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
|
||||
--run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
|
||||
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
||||
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
||||
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
||||
@@ -239,7 +239,7 @@ gen_status() {
|
||||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||||
--tests "${CONF_ROOT}/tests" \
|
||||
--out "${STATUS_OUT}" \
|
||||
--env "${CONF_ENV:-dev}" \
|
||||
${CONF_ENV:+--env "$CONF_ENV"} \
|
||||
"$@"
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="dr-restore" classname="" time="150.698290" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,672 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/dr-restore
|
||||
=== PAUSE chainsaw/dr-restore
|
||||
=== CONT chainsaw/dr-restore
|
||||
l.go:52: | 02:15:05 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-glorious-sheep
|
||||
l.go:52: | 02:15:05 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:15:05 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
|
||||
# ---- helpers -------------------------------------------------------
|
||||
# Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
|
||||
# fail loudly; an absent API is a genuine precondition gap and may skip.
|
||||
die_or_skip() {
|
||||
case "$2" in
|
||||
*orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
|
||||
echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
|
||||
echo "$2" >&2
|
||||
exit 1 ;;
|
||||
*)
|
||||
echo "SKIP: $1 not available on this cluster -- $2"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
ip2int() {
|
||||
_q="$(echo "$1" | tr '.' ' ')"
|
||||
set -- $_q
|
||||
[ $# -eq 4 ] || { echo -1; return 0; }
|
||||
echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
|
||||
}
|
||||
|
||||
# in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
|
||||
in_cidr() {
|
||||
case "$2" in *"/"*) ;; *) return 1 ;; esac
|
||||
_net="${2%/*}"; _len="${2#*/}"
|
||||
case "$_len" in ''|*[!0-9]*) return 1 ;; esac
|
||||
[ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
|
||||
_i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
|
||||
[ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
|
||||
[ "$_len" -eq 0 ] && return 0
|
||||
_d=1; _s=0
|
||||
while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
|
||||
[ $(( _i / _d )) -eq $(( _n / _d )) ]
|
||||
}
|
||||
|
||||
# ---- Gate 1: the replicated storage tier the DR-critical data lives on --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
|
||||
2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
|
||||
if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
|
||||
echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "backup storage locations:"
|
||||
echo "$BSL_LINES"
|
||||
|
||||
VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
|
||||
|
||||
# ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
|
||||
# File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
|
||||
# deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
|
||||
# ready node-agent no volume data can be captured, so the drill is not runnable.
|
||||
VREADY=0; NAREADY=0
|
||||
for i in $(seq 1 12); do
|
||||
VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
|
||||
NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
|
||||
echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
|
||||
if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
|
||||
sleep 10
|
||||
done
|
||||
if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
|
||||
echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Pick the BSL this drill will use: prefer the default, and it MUST be
|
||||
# Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
|
||||
# gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
|
||||
# wait for first-validation on a cold start), it does not skip. --------
|
||||
BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
|
||||
for i in $(seq 1 12); do
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -n "$CAND" ]; then
|
||||
BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
|
||||
BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
|
||||
break
|
||||
fi
|
||||
echo "poll $i: no Available BackupStorageLocation yet"
|
||||
sleep 10
|
||||
done
|
||||
if [ -z "$BSL_NAME" ]; then
|
||||
echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
|
||||
echo " The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
|
||||
echo "$BSL_LINES" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "using BSL ${BSL_NS}/${BSL_NAME} bucket=${BSL_BUCKET} endpoint=${BSL_URL}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
|
||||
if [ -z "$BSL_URL" ]; then
|
||||
echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
|
||||
echo " the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
|
||||
echo " test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
|
||||
exit 1
|
||||
fi
|
||||
HOST="${BSL_URL#*://}" # strip scheme
|
||||
HOST="${HOST%%/*}" # strip path
|
||||
HOST="${HOST%%:*}" # strip port
|
||||
echo "backup endpoint host: $HOST"
|
||||
|
||||
case "$HOST" in
|
||||
localhost|*.svc|*.svc.cluster.local|*.cluster.local)
|
||||
echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
|
||||
echo " An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
|
||||
# Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
|
||||
# off-cluster property -- that is a failure, never a silent pass.
|
||||
case "$HOST" in
|
||||
*[!0-9.]*)
|
||||
# Endpoint given as a name. Try the resolvers that may exist in the
|
||||
# runner image, in order; if none is available or none resolves, the
|
||||
# test FAILS below rather than guessing.
|
||||
EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
|
||||
*) EPIP="$HOST" ;;
|
||||
esac
|
||||
if [ -z "$EPIP" ]; then
|
||||
echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
|
||||
echo " the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "backup endpoint resolves to: $EPIP"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
|
||||
case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
|
||||
|
||||
NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for a in $NODE_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
|
||||
|
||||
SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $SVC_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
|
||||
|
||||
POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $POD_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
|
||||
|
||||
POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for c in $POD_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac # IPv6 prefixes are out of scope here
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
|
||||
|
||||
# Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
|
||||
# runner may not read): the checks above are the authoritative ones.
|
||||
SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
if [ -n "$SVC_CIDRS" ]; then
|
||||
for c in $SVC_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
|
||||
else
|
||||
echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
|
||||
fi
|
||||
|
||||
echo "OFF_CLUSTER_TARGET_OK bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
backup storage locations:
|
||||
velero default Available aws true https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com k3s-prod-backup
|
||||
poll 1: velero readyReplicas=1 node-agent numberReady=3
|
||||
using BSL velero/default bucket=k3s-prod-backup endpoint=https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
|
||||
backup endpoint host: 7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
|
||||
backup endpoint resolves to: 172.64.66.1
|
||||
endpoint is not any node address (checked 6 node addresses)
|
||||
endpoint is not any Service ClusterIP (checked 82)
|
||||
endpoint is not any Pod IP (checked 197)
|
||||
endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
|
||||
endpoint is outside every ServiceCIDR (10.43.0.0/16 )
|
||||
OFF_CLUSTER_TARGET_OK bsl=velero/default endpoint=172.64.66.1 bucket=k3s-prod-backup
|
||||
l.go:52: | 02:15:08 | [32;1mdr-restore[0;22m | [32;1massert-backup-target-is-off-cluster [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
SCHED="$(kubectl get schedules.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
|
||||
# Distinguish "Velero absent" (skip, in lock-step with step 1) from
|
||||
# "Velero present but nothing scheduled" (a real R-STATE-03 defect).
|
||||
if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
|
||||
if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
|
||||
echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
|
||||
echo " R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "velero schedules:"
|
||||
echo "$SCHED"
|
||||
# A Schedule that exists but is paused/disabled backs nothing up.
|
||||
if echo "$SCHED" | grep -q 'phase=Enabled'; then
|
||||
echo BACKUPS_ARE_SCHEDULED_OK
|
||||
else
|
||||
echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
|
||||
exit 1
|
||||
fi
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero schedules:
|
||||
velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T04:01:06Z
|
||||
velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T03:00:03Z
|
||||
BACKUPS_ARE_SCHEDULED_OK
|
||||
l.go:52: | 02:15:08 | [32;1mdr-restore[0;22m | [32;1massert-backups-are-scheduled [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:15:08 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src" # scratch source namespace (destroyed by design)
|
||||
DST="${NS}-dr-dst" # clean restore target (must not pre-exist)
|
||||
BK="${NS}-dr-backup"
|
||||
RS="${NS}-dr-restore"
|
||||
OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
|
||||
|
||||
# ---- Re-derive the preconditions (steps do not share shell state). Every
|
||||
# skip below mirrors step 1 exactly, so the whole test skips or runs
|
||||
# as a unit -- it can never half-run. -----------------------------
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -z "$CAND" ]; then
|
||||
echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
VNS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
|
||||
echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
|
||||
|
||||
# A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
|
||||
# be free. If either is lying around from an aborted run, remove it before we
|
||||
# claim "clean" -- and fail loudly if it will not go, because a half-deleted
|
||||
# namespace would silently invalidate the destroy proof below.
|
||||
kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
|
||||
|
||||
# ============================ GIVEN ==============================
|
||||
# A DR-critical stateful workload with known data on the replicated tier.
|
||||
CANARY="dr-canary-$(date +%s%N)"
|
||||
kubectl create namespace "$SRC"
|
||||
kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: dr-data
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: dr-workload
|
||||
labels:
|
||||
app: dr-workload
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 7200"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: dr-data
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
|
||||
printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
|
||||
echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
|
||||
|
||||
ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
|
||||
[ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
|
||||
echo "original in-cluster PersistentVolume: $ORIG_PV"
|
||||
|
||||
# ======================= WHEN (part 1): back up off-cluster =======
|
||||
# Created as a Velero Backup CR -- exactly what `velero backup create` does.
|
||||
# defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
|
||||
# the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
|
||||
# ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
|
||||
# own GC still removes this drill's objects from the bucket.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: ${BK}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
includedNamespaces: [${SRC}]
|
||||
storageLocation: ${BSL_NAME}
|
||||
snapshotVolumes: false
|
||||
defaultVolumesToFsBackup: true
|
||||
ttl: 2h0m0s
|
||||
EOF
|
||||
|
||||
BPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: backup phase=${BPHASE:-<pending>}"
|
||||
case "${BPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${BPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "backup $BK Completed to ${BSL_NAME}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
|
||||
# kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
|
||||
PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
|
||||
-l velero.io/backup-name="$BK" \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
|
||||
echo "pod volume backups:"
|
||||
echo "$PVB"
|
||||
if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
|
||||
echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
|
||||
echo " Resource-only backups cannot restore DR-critical DATA." >&2
|
||||
exit 1
|
||||
fi
|
||||
# A line is "<name> <phase> <snapshotID>"; require Completed AND a
|
||||
# non-empty snapshot ID (an empty ID means nothing reached the repository).
|
||||
GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
|
||||
if [ "${GOOD_PVB:-0}" -lt 1 ]; then
|
||||
echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
|
||||
echo " the volume data did not reach the off-cluster repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
|
||||
|
||||
# ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
|
||||
# Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
|
||||
# underlying PersistentVolume (and with it the Longhorn replicas). After this
|
||||
# there is no in-cluster source the restore could quietly read from.
|
||||
kubectl delete namespace "$SRC" --wait=true --timeout=420s
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for namespace $SRC to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get namespace "$SRC" >/dev/null 2>&1; then
|
||||
echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "source namespace destroyed"
|
||||
|
||||
# The longhorn StorageClass reclaims Delete, so the PV should already be gone;
|
||||
# force it if a Retain policy (or a stuck finalizer) left it behind, because a
|
||||
# surviving PV would make the restore prove nothing about off-cluster recovery.
|
||||
kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
|
||||
echo " An in-cluster copy of the data still exists, so a successful restore" >&2
|
||||
echo " would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
|
||||
|
||||
# Best-effort corroboration (the least-privilege runner may not read longhorn.io):
|
||||
# the Longhorn Volume CR and its replicas should be gone too.
|
||||
if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
|
||||
|
||||
# ============================ RESTORE ============================
|
||||
# Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
|
||||
# the catalog's "restored into a clean state".
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Restore
|
||||
metadata:
|
||||
name: ${RS}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
includedNamespaces: [${SRC}]
|
||||
namespaceMapping:
|
||||
${SRC}: ${DST}
|
||||
existingResourcePolicy: none
|
||||
EOF
|
||||
|
||||
RPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: restore phase=${RPHASE:-<pending>}"
|
||||
case "${RPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${RPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "restore $RS Completed into $DST"
|
||||
|
||||
# ============================= THEN ==============================
|
||||
# The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
|
||||
# `restore-wait` init container that blocks until the node-agent has written the
|
||||
# volume contents back, so Ready here already implies the data landed.
|
||||
if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
|
||||
echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
|
||||
kubectl -n "$DST" describe pod dr-workload >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
|
||||
[ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
|
||||
if [ "$NEW_PV" = "$ORIG_PV" ]; then
|
||||
echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
|
||||
echo " That is an in-cluster survival, not an off-cluster restore." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
|
||||
|
||||
GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
if [ "$GOT" != "$CANARY" ]; then
|
||||
echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero ns=velero bsl=default src=chainsaw-glorious-sheep-dr-src dst=chainsaw-glorious-sheep-dr-dst
|
||||
namespace/chainsaw-glorious-sheep-dr-src created
|
||||
namespace/chainsaw-glorious-sheep-dr-src labeled
|
||||
persistentvolumeclaim/dr-data created
|
||||
pod/dr-workload created
|
||||
pod/dr-workload condition met
|
||||
seeded canary: dr-canary-1789352108935754215
|
||||
original in-cluster PersistentVolume: pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9
|
||||
backup.velero.io/chainsaw-glorious-sheep-dr-backup created
|
||||
poll 1: backup phase=InProgress
|
||||
poll 2: backup phase=InProgress
|
||||
poll 3: backup phase=InProgress
|
||||
poll 4: backup phase=Completed
|
||||
backup chainsaw-glorious-sheep-dr-backup Completed to default
|
||||
pod volume backups:
|
||||
chainsaw-glorious-sheep-dr-backup-2gjrq Completed 979eed437233b45c7f85ba80074ab56b
|
||||
volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
|
||||
namespace "chainsaw-glorious-sheep-dr-src" deleted
|
||||
source namespace destroyed
|
||||
original PersistentVolume pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 is GONE -- no in-cluster copy of the data remains
|
||||
(longhorn Volume CR pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 not present -- either deleted or not readable by this identity)
|
||||
restore.velero.io/chainsaw-glorious-sheep-dr-restore created
|
||||
poll 1: restore phase=InProgress
|
||||
poll 2: restore phase=InProgress
|
||||
poll 3: restore phase=InProgress
|
||||
poll 4: restore phase=Completed
|
||||
restore chainsaw-glorious-sheep-dr-restore Completed into chainsaw-glorious-sheep-dr-dst
|
||||
pod/dr-workload condition met
|
||||
restored PVC bound to PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
|
||||
restored onto a NEW volume (pvc-f0c061e1-c080-49f2-a381-23da5c886df2 != pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9) -- the data came from outside the cluster
|
||||
recovered='dr-canary-1789352108935754215' expected='dr-canary-1789352108935754215'
|
||||
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
=== STDERR
|
||||
Defaulted container "app" out of: app, restore-wait (init)
|
||||
l.go:52: | 02:17:18 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:17:18 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -u
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
|
||||
|
||||
VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
|
||||
|
||||
# Velero objects first: a DeleteBackupRequest makes Velero remove the backup
|
||||
# DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
|
||||
if [ -n "$VNS" ]; then
|
||||
kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
|
||||
if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
|
||||
kubectl apply -f - <<EOF || true
|
||||
apiVersion: velero.io/v1
|
||||
kind: DeleteBackupRequest
|
||||
metadata:
|
||||
name: ${BK}-delete
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
EOF
|
||||
for i in $(seq 1 18); do
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
|
||||
sleep 10
|
||||
done
|
||||
# Backstop if Velero could not process the request (the 2h ttl also GCs it).
|
||||
kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
|
||||
kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
|
||||
fi
|
||||
echo "velero objects for $NS cleaned up (ns $VNS)"
|
||||
fi
|
||||
|
||||
# Scratch namespaces (both, in case the run aborted at any point).
|
||||
for n in "$DST" "$SRC"; do
|
||||
kubectl delete namespace "$n" --ignore-not-found --wait=false || true
|
||||
done
|
||||
# Anything left labelled as ours (defensive; the two names above are the only
|
||||
# namespaces this test ever creates).
|
||||
kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
|
||||
|
||||
# Released PVs from either namespace's PVCs would otherwise leak storage.
|
||||
for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
|
||||
kubectl delete pv "$pv" --ignore-not-found --wait=false || true
|
||||
echo "released leftover PersistentVolume: $pv"
|
||||
done
|
||||
|
||||
echo "dr-restore teardown complete (best-effort; every object was created by this test)"
|
||||
l.go:52: | 02:17:30 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
restore.velero.io "chainsaw-glorious-sheep-dr-restore" deleted from velero namespace
|
||||
deletebackuprequest.velero.io/chainsaw-glorious-sheep-dr-backup-delete created
|
||||
velero objects for chainsaw-glorious-sheep cleaned up (ns velero)
|
||||
namespace "chainsaw-glorious-sheep-dr-dst" deleted
|
||||
namespace "chainsaw-glorious-sheep-dr-dst" deleted
|
||||
persistentvolume "pvc-f0c061e1-c080-49f2-a381-23da5c886df2" deleted
|
||||
released leftover PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
|
||||
dr-restore teardown complete (best-effort; every object was created by this test)
|
||||
l.go:52: | 02:17:30 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 02:17:30 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 02:17:30 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 02:17:30 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-glorious-sheep
|
||||
l.go:52: | 02:17:36 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/dr-restore (150.70s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,13 @@
|
||||
<?xml version="1.0" ?>
|
||||
<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
|
||||
<testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
|
||||
<testcase classname="L0:R-ENV-02,R-STATE-03" name="placeholder-guard-coupling">
|
||||
<properties>
|
||||
<property name="requirements" value="R-ENV-02,R-STATE-03"/>
|
||||
<property name="layer" value="L0"/>
|
||||
<property name="path" value="/work/conformance/tests/l0-static/placeholder-guard-coupling"/>
|
||||
<property name="scope" value=""/>
|
||||
</properties>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,7 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m Running 1 offline L0/L1 check(s) (no cluster; docs/14).
|
||||
[offline] L0 PASS placeholder-guard-coupling (R-ENV-02,R-STATE-03)
|
||||
[offline] wrote JUnit (1 case(s), 0 failure(s), 0 skip(s), 0 unwired) -> /work/conformance/_artifacts/report.junit.xml
|
||||
[1m[run.sh][0m Done (offline l0). JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,6 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
|
||||
Install the pinned binary or run via the in-cluster Job (incluster/).
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="115.855688" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,232 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 05:22:26 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1789795346371286043
|
||||
victim node (currently hosts the attached volume): k3s-prod-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-prod-2 labeled
|
||||
node/k3s-prod-2 cordoned
|
||||
cordoned + labelled victim node: k3s-prod-2
|
||||
pod "writer" deleted from chainsaw-advanced-boxer namespace
|
||||
evicted writer off the victim node; volume detached from k3s-prod-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
|
||||
recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-prod-2 uncordoned
|
||||
node/k3s-prod-2 unlabeled
|
||||
restored node to service: k3s-prod-2
|
||||
l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-advanced-boxer namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-advanced-boxer
|
||||
l.go:52: | 05:24:21 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (115.86s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -3,3 +3,6 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}
|
||||
@@ -0,0 +1,42 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
|
||||
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
|
||||
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
|
||||
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
|
||||
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
|
||||
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
|
||||
#
|
||||
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
|
||||
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
|
||||
# would have skipped this check on the very commit that introduced the defect.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: admission-policies-actually-applied
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
|
||||
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
|
||||
exemption that is genuinely not applied — so a policy cannot be authored, tested,
|
||||
green, and absent from the cluster.
|
||||
spec:
|
||||
steps:
|
||||
- name: every-authored-policy-is-applied-or-documented-ci-only
|
||||
description: >-
|
||||
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
|
||||
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — admission-policies-actually-applied
|
||||
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
|
||||
#
|
||||
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
|
||||
#
|
||||
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
|
||||
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
|
||||
# "kept as reference; THIS file is the buildable render root".
|
||||
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
|
||||
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
|
||||
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
|
||||
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
|
||||
#
|
||||
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
|
||||
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
|
||||
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
|
||||
# `platform` Ready at the new revision — and the policy was never in any applied set. The
|
||||
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
|
||||
# which is precisely the state that identity's own RBAC header says must never exist.
|
||||
#
|
||||
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
|
||||
# DEPLOYED. This check asserts the second thing.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
|
||||
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
|
||||
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
|
||||
# policy that IS applied, so the exemption list cannot rot into a lie either way.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import sys, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
|
||||
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
|
||||
# that silently starts being applied is as much a lie as one that silently stops.
|
||||
CI_ONLY = {
|
||||
"disallow-plaintext-secrets":
|
||||
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
|
||||
"admission because Flux decrypts before the Secret reaches the API server. Its "
|
||||
"in-cluster counterpart is admission/secret-provenance.yaml.",
|
||||
}
|
||||
|
||||
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
|
||||
"platform/kyverno/*.yaml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
|
||||
|
||||
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
|
||||
policies = {}
|
||||
for f in files:
|
||||
try:
|
||||
txt = f.read_text()
|
||||
except Exception:
|
||||
continue
|
||||
if "kind: ClusterPolicy" not in txt:
|
||||
continue
|
||||
for line in txt.splitlines():
|
||||
st = line.strip()
|
||||
if st.startswith("name:") and " name:" in line:
|
||||
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
|
||||
break
|
||||
|
||||
check(len(policies) >= 6,
|
||||
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
|
||||
f"too few means the scan broke, and every assertion below would be vacuous")
|
||||
|
||||
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
|
||||
|
||||
unapplied, wrongly_exempt = [], []
|
||||
for rel, name in sorted(policies.items()):
|
||||
# platform-config references these as ../platform/kyverno/...
|
||||
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
|
||||
if name in CI_ONLY:
|
||||
if referenced:
|
||||
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
|
||||
elif not referenced:
|
||||
unapplied.append(f"{name} ({rel})")
|
||||
|
||||
check(not unapplied,
|
||||
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
|
||||
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
|
||||
|
||||
check(not wrongly_exempt,
|
||||
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
|
||||
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
|
||||
|
||||
for name, reason in CI_ONLY.items():
|
||||
check(bool(reason.strip()),
|
||||
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
|
||||
f"list stops being reviewable")
|
||||
|
||||
check(CHECKS >= 4,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
|
||||
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
|
||||
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
|
||||
f"not that they are correct — `kyverno test` is what proves correctness.")
|
||||
PY
|
||||
@@ -36,18 +36,27 @@
|
||||
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
|
||||
# check above can never see this failure: where the repo authors a Service and a
|
||||
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
|
||||
# actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
|
||||
# port `metrics`; no app Service defines one), so they are registered in
|
||||
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
|
||||
# actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
|
||||
# named port `metrics`; no app Service defined one) and were registered in
|
||||
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
|
||||
# THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
|
||||
# kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
|
||||
# ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
|
||||
# moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
|
||||
# alert rule to reach a monitor, resolves by selector across the whole tree instead of by
|
||||
# same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
|
||||
# lack of subjects;
|
||||
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
|
||||
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
|
||||
#
|
||||
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
|
||||
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
|
||||
# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
|
||||
# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
|
||||
# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
||||
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
|
||||
# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
|
||||
# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
|
||||
# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
||||
# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
|
||||
# signal that already existed. The register is empty now, and that is the outcome it was for.)
|
||||
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
|
||||
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
|
||||
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
|
||||
|
||||
@@ -182,14 +182,42 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
|
||||
"prefixes": ("kubelet_",),
|
||||
# container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as
|
||||
# kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix
|
||||
# was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it).
|
||||
# Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the
|
||||
# three nodes, job="kubelet", metrics_path="/metrics/cadvisor".
|
||||
"prefixes": ("kubelet_", "container_"),
|
||||
"file": KPS,
|
||||
# Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
|
||||
# honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
|
||||
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
|
||||
"forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
|
||||
"pvc-capacity-alerts rules",
|
||||
"pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the "
|
||||
"only signal on this cluster that can name WHICH workload an I/O stall is hitting",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT "
|
||||
"disabled the way kubeEtcd/kubeProxy are)",
|
||||
# DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped
|
||||
# metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's
|
||||
# server-side disk metrics (etcd_disk_wal_fsync_duration_seconds,
|
||||
# etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for
|
||||
# k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for
|
||||
# {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket
|
||||
# `etcd_` claim would let a future rule on those names pass this check while being inert,
|
||||
# which is precisely the hole this test exists to close. Widen only with a real scrape.
|
||||
"exact": (
|
||||
"etcd_request_duration_seconds_bucket",
|
||||
"etcd_request_duration_seconds_count",
|
||||
),
|
||||
"file": KPS,
|
||||
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
|
||||
"forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path "
|
||||
"this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the "
|
||||
"condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
|
||||
@@ -228,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"metricLabelsAllowlist:",
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
|
||||
# The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
|
||||
# end in a literal `]`, asserting the list held exactly this one label — so adding a
|
||||
# second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
|
||||
# while the property it guards stayed perfectly satisfied. A guard that breaks when you
|
||||
# correctly extend the thing it guards teaches people to edit the guard, which is how it
|
||||
# stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
|
||||
@@ -333,38 +367,36 @@ KNOWN_UNWIRED_METRICS: dict[str, str] = {
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
|
||||
# names an endpoint port that the Service it selects does not define, so the monitor discovers
|
||||
# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
|
||||
# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
|
||||
# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
|
||||
# has no up series to be 0.
|
||||
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only, and NOW EMPTY.
|
||||
#
|
||||
# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
|
||||
# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
|
||||
# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
|
||||
# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
|
||||
# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
|
||||
# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
|
||||
# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
|
||||
# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
|
||||
# Each entry was a `up{job="X"}` selector whose monitor named an endpoint port the Service it
|
||||
# selects did not define, so the monitor discovered ZERO targets, no `up{job="X"}` series existed,
|
||||
# and `up{job="X"} == 0` could never fire. The nastiest variant of the class: `up` exists in
|
||||
# abundance, so nothing about the metric name looks wrong, and `PrometheusTargetsMissing` cannot see
|
||||
# it either — a target that was never discovered has no up series to be 0.
|
||||
#
|
||||
# IT HELD ALL TWELVE apps/* SLI RULES, AND IT IS EMPTY BECAUSE THEY WERE FIXED, not because the gap
|
||||
# was waived (2026-09-13, branch fix/app-sli-rules-discover-no-targets). The register's own note
|
||||
# said closing it needed "a Prometheus exporter per app". That was the wrong conclusion from a true
|
||||
# premise: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud
|
||||
# genuinely export no Prometheus metrics, so instead the `up` SLI was rewritten against a signal that
|
||||
# DOES exist for every workload — kube-state-metrics readiness of the service's own workload object
|
||||
# (`kube_deployment_status_replicas_available` / `kube_statefulset_status_replicas_ready`), and
|
||||
# `kubevirt_vmi_info{phase="running"}` for the VM. No exporter, no new scrape target; each rule is
|
||||
# unit-tested firing AND quiet (apps/wave-*/tests/app-slis.test.yaml) and each expression was
|
||||
# evaluated live on prod, where eleven of the twelve return no series on a healthy cluster and the
|
||||
# twelfth (vm-demo) is not deployed at all.
|
||||
#
|
||||
# The rendered ServiceMonitors went with them: a monitor for a workload that serves no metrics is
|
||||
# dead configuration that reads as coverage. So assertion (4) below now has NOTHING to audit under
|
||||
# apps/ — which is why the monitor-side axis moved into its own check,
|
||||
# tests/l0-static/servicemonitor-ports-resolve. THAT is where a ServiceMonitor naming a port no
|
||||
# Service defines fails, with no alert rule required, resolving by selector across the whole tree
|
||||
# rather than by same-file adjacency, and with its resolver proven on fixtures so it cannot pass for
|
||||
# lack of subjects. Keeping this register non-empty as "documentation" of a closed gap is exactly the
|
||||
# fiction the ratchet forbids: the stale-entry assertion below would fail, and rightly.
|
||||
# =============================================================================================
|
||||
KNOWN_UNWIRED_JOBS: dict[str, str] = {
|
||||
"gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
|
||||
"jackett": "apps/wave-0/jackett.yaml — no exporter",
|
||||
"pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
|
||||
"plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
|
||||
"qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
|
||||
"vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
|
||||
"keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
|
||||
"keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
|
||||
"neither, so it is a rename away from being live rather than an exporter away",
|
||||
"nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
|
||||
"radarr": "apps/wave-1/radarr.yaml — needs exportarr",
|
||||
"semaphore": "apps/wave-1/semaphore.yaml — no exporter",
|
||||
"sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
|
||||
"seerr": "apps/wave-2/seerr.yaml — no exporter",
|
||||
}
|
||||
KNOWN_UNWIRED_JOBS: dict[str, str] = {}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) PARSE EVERY ALERT RULE
|
||||
@@ -422,9 +454,14 @@ def is_golden_fixture(p: Path) -> bool:
|
||||
return "conformance" in parts and "golden" in parts
|
||||
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
@@ -593,7 +630,7 @@ def yaml_docs(path: Path) -> list[dict]:
|
||||
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
|
||||
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts:
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
|
||||
continue
|
||||
txt = read(p)
|
||||
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
|
||||
|
||||
@@ -147,25 +147,20 @@ def check(cond, msg):
|
||||
# =============================================================================
|
||||
# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
|
||||
#
|
||||
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
|
||||
# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
|
||||
# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
|
||||
# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
|
||||
# FAILS as stale.
|
||||
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
|
||||
# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
|
||||
#
|
||||
# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
|
||||
# generated from one ServiceDefinition template, and the register's description of them as one
|
||||
# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
|
||||
# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
|
||||
# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
|
||||
# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
|
||||
# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
|
||||
# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
|
||||
# now the only entry left and is where the next tests belong.
|
||||
# =============================================================================
|
||||
UNTESTED_FILES = {
|
||||
"apps/wave-0/gitea.yaml",
|
||||
"apps/wave-0/jackett.yaml",
|
||||
"apps/wave-0/pihole.yaml",
|
||||
"apps/wave-0/plex.yaml",
|
||||
"apps/wave-0/qbittorrent.yaml",
|
||||
"apps/wave-0/vm-demo.yaml",
|
||||
"apps/wave-1/keycloak.yaml",
|
||||
"apps/wave-1/nextcloud.yaml",
|
||||
"apps/wave-1/radarr.yaml",
|
||||
"apps/wave-1/semaphore.yaml",
|
||||
"apps/wave-1/sonarr.yaml",
|
||||
"apps/wave-2/seerr.yaml",
|
||||
"platform/kubevirt/alerts.yaml",
|
||||
}
|
||||
|
||||
@@ -173,8 +168,21 @@ UNTESTED_FILES = {
|
||||
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
|
||||
# already-"tested" file untested: anything not listed here must appear in a covering test.
|
||||
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
|
||||
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
|
||||
# rules file by the suite convention but has no assertions of its own yet.
|
||||
# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
|
||||
# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
|
||||
# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
|
||||
# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
|
||||
# redundancy meant asserting on the rules that survive). Only
|
||||
# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
|
||||
# It SHRANK BY TWO MORE on 2026-09-14: platform/monitoring/tests/node-io-cause.test.yaml pins the
|
||||
# eight new platform-health.node-io-cause rules, and its healthy fixture asserts NodeIOPressureHigh
|
||||
# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
|
||||
# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
|
||||
# detectors are no longer unasserted.
|
||||
# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
|
||||
# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
|
||||
# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
|
||||
# had to become an assertion rather than a sentence in a comment.
|
||||
UNTESTED_ALERTS = {
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
||||
@@ -186,30 +194,20 @@ UNTESTED_ALERTS = {
|
||||
("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
|
||||
("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
|
||||
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
|
||||
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
||||
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
||||
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
|
||||
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
|
||||
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
|
||||
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
|
||||
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
|
||||
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
|
||||
}
|
||||
|
||||
@@ -285,11 +283,16 @@ FETCH_RE = re.compile(
|
||||
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
|
||||
# version, the network and the machine are decided silently at test time, and a missing
|
||||
# toolchain becomes a green run instead of an exit 3.
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
|
||||
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
|
||||
for p in sorted(ROOT.rglob("*.sh")):
|
||||
rel = str(p.relative_to(ROOT))
|
||||
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
|
||||
if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
|
||||
continue
|
||||
try:
|
||||
text = p.read_text()
|
||||
@@ -324,7 +327,7 @@ def tracked_yaml() -> list[Path]:
|
||||
return [ROOT / p for p in out.stdout.split("\0") if p]
|
||||
# No usable index (the gate container mounts a worktree whose .git points outside the
|
||||
# mount). Fall back to a filesystem walk rather than auditing nothing.
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
|
||||
|
||||
|
||||
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
|
||||
@@ -478,7 +481,34 @@ for rel in sorted(alert_files):
|
||||
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
|
||||
f"shrink-only: delete the line so the next missing test is caught")
|
||||
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
|
||||
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
|
||||
# "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
|
||||
# This used to be a bare word match over the whole test file, so an alert named anywhere —
|
||||
# including in a PROSE COMMENT — counted as tested. Hit for real while adding
|
||||
# platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
|
||||
# stays quiet during an egress-steering failure, and that sentence alone was enough to make this
|
||||
# check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
|
||||
# the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
|
||||
# outcome this check exists to prevent, produced BY the check.
|
||||
# Now an alert counts only when a test file names it in an `alertname:` position, with comment
|
||||
# lines stripped first so a commented-out assertion does not count either.
|
||||
# MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
|
||||
# asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
|
||||
# registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
|
||||
# mattered because the registers are shrink-only: had any alert been prose-only, tightening would
|
||||
# have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
|
||||
# it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
|
||||
text_no_comments = "\n".join(
|
||||
l for l in text.split("\n") if not l.lstrip().startswith("#")
|
||||
)
|
||||
# BOTH promtool idioms count, because both are real assertions:
|
||||
# alert_rule_test: - alertname: CNPGPrimariesCoLocated
|
||||
# promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
# Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
|
||||
# platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
|
||||
# form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
|
||||
# already better covered than most.
|
||||
covered = [n for n in names
|
||||
if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
|
||||
covered_total += len(covered)
|
||||
for n in names:
|
||||
if n in covered:
|
||||
|
||||
@@ -126,9 +126,14 @@ def read(p: Path) -> str:
|
||||
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
|
||||
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
|
||||
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the C2 root surfaces every service the cluster runs, plus platform
|
||||
# health and controlled actions (R-C2-01, R-C2-02) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE REQUIREMENTS, VERBATIM (spec docs/00-vision-and-scope.md):
|
||||
# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
|
||||
# running in the cluster and links to them."
|
||||
# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
|
||||
# provide controlled operational actions where safe."
|
||||
# Detail: spec docs/12-command-center-ui.md — Homepage at the root is "the primary R-C2-01
|
||||
# surface"; Headlamp is "the R-C2-02 'controlled operational actions where safe' surface";
|
||||
# Grafana is the dashboards/alerts surface; "never a raw cluster-admin kubeconfig in a browser".
|
||||
#
|
||||
# WHY L0, AND NOT L2
|
||||
# ------------------
|
||||
# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
|
||||
# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
|
||||
# verifies the certificate it serves. What no live probe can decide is the half the
|
||||
# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
|
||||
# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
|
||||
# what git declares the cluster serves (every in-repo HTTPRoute hostname, every L4 Service
|
||||
# that publishes a hostname, every catalog ServiceDefinition) against what the dashboard's
|
||||
# ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides half the
|
||||
# estate and cannot tell the difference.
|
||||
#
|
||||
# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
|
||||
# service, forget the dashboard, and every cluster test stays green while the operator's
|
||||
# front door hides a running service. That is not a styling preference — it is exactly the
|
||||
# R-C2-01 failure mode, and this check found a live instance of it when it was written (see
|
||||
# the commit message: platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root
|
||||
# linked to it).
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the apex ingress, the dashboard's declared link set, every in-repo HTTPRoute, every
|
||||
# hostname-publishing L4 Service, the versioned catalog and the platform-health
|
||||
# surfaces are read and compared,
|
||||
# THEN — R-C2-01 —
|
||||
# the bare apex is claimed by exactly one route, that route binds the ONE Gateway
|
||||
# listener whose hostname matches a bare apex, the apex is a SAN on the certificate
|
||||
# that listener terminates with, the dashboard is RECONCILED (not merely authored),
|
||||
# FORWARD every HTTP service the repo routes is linked from the root, FORWARD every
|
||||
# catalog `kind: container` service with a route is linked, and REVERSE every
|
||||
# home-zone link the root offers resolves to something this repo actually serves —
|
||||
# an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4);
|
||||
# — R-C2-02 —
|
||||
# both health surfaces docs/12 names are linked AND reconciled, the health DATA exists
|
||||
# as code (PrometheusRule groups for cluster/nodes, storage, GitOps and cert expiry,
|
||||
# reconciled by the platform-config tier), and actions are CONTROLLED: the console
|
||||
# binds NO ClusterRole to its own ServiceAccount, authenticates per-user by native
|
||||
# OIDC from a named SOPS Secret, and cluster rights arrive through OIDC GROUPS at more
|
||||
# than one privilege level.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files + PyYAML —
|
||||
# the same dependency catalog/render.py already requires). conformance/run.sh --layer l0 and
|
||||
# ci/l0.sh run it directly; this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# WHAT IT DOES NOT PROVE — honest gaps, restated in check.sh's own header:
|
||||
# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2;
|
||||
# the Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
|
||||
# - That the C2 is access-CONTROLLED in fact. The auth assertions read a DECLARED posture,
|
||||
# because the Gateway ext-authz attachment that would enforce forward-auth is not built
|
||||
# anywhere in this repo (platform/homepage/httproute.yaml says so in its own header, and
|
||||
# every forward-auth app shares the gap). Declared, not enforced — and labelled as such
|
||||
# in the assertion text.
|
||||
# - That an operational action in the console succeeds, or that it is audited. The
|
||||
# assertions prove there is no shared cluster-admin identity to act with and that rights
|
||||
# arrive per-user; the audit trail is a runtime property.
|
||||
# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
|
||||
# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: c2-root-surface
|
||||
labels:
|
||||
conformance.home.k3s/R-C2-01: "true"
|
||||
conformance.home.k3s/R-C2-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-C2-01, R-C2-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE, and it is the whole point of the forward assertion.
|
||||
#
|
||||
# The invariant is "every service the repo routes is linked from the C2 root". Its input
|
||||
# surface is therefore EVERY tracked manifest that can declare an HTTPRoute — apps/wave-*,
|
||||
# platform/*, runbooks/ — not just the two components that own the C2. ci/touched-
|
||||
# requirements.sh keys `platform/homepage/*` and `platform/headlamp/*` to nothing at all
|
||||
# today (there is no path-map arm for either), and even with one, the change that BREAKS
|
||||
# this invariant is the change that adds a route somewhere else: `apps/wave-1/newthing.yaml`
|
||||
# attributes R-GITOPS-02 / R-NET-* / nothing, never R-C2-01. Requirement-scoped selection
|
||||
# would filter this check out of exactly the change that introduces the bug — the R-TEST-03
|
||||
# blind spot recorded in conformance/lib/common.sh.
|
||||
#
|
||||
# Declaring the scope is the whole fix: conformance/run.sh --include-scope whole-repo
|
||||
# (ci/l0.sh Stage 3b) unions this test into EVERY gate invocation. Do NOT hand-add a stage
|
||||
# to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
# A SKIP of a scope-declaring check is rc 3 (UNWIRED), never green.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
The bare apex is bound to the landing dashboard over trusted TLS and is reconciled; the
|
||||
dashboard links every HTTP service this repo routes and every link it offers resolves to
|
||||
something the repo serves; and the C2 surfaces platform health backed by real alert rules
|
||||
while granting operational rights per-user through OIDC groups rather than through a
|
||||
shared cluster-admin identity.
|
||||
spec:
|
||||
steps:
|
||||
- name: compare-the-c2-root-against-what-the-repo-actually-serves
|
||||
description: >-
|
||||
Read the apex ingress chain, the dashboard's declared links, every in-repo HTTPRoute and
|
||||
hostname-publishing Service, the catalog, the platform-health alert rules and the
|
||||
console's RBAC posture, and assert the two sets agree in both directions.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+689
@@ -0,0 +1,689 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — c2-root-surface
|
||||
#
|
||||
# Serves (spec docs/00-vision-and-scope.md; detail in docs/12-command-center-ui.md):
|
||||
#
|
||||
# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the
|
||||
# services running in the cluster and links to them."
|
||||
# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps,
|
||||
# alerts) and provide controlled operational actions where safe."
|
||||
#
|
||||
# WHY THIS IS AN L0 STATIC CHECK AND NOT AN L2 PROBE
|
||||
# --------------------------------------------------
|
||||
# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
|
||||
# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
|
||||
# verifies the certificate it serves. What no live probe can decide is the half the
|
||||
# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
|
||||
# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
|
||||
# what git declares the cluster serves (every in-repo HTTPRoute hostname, every
|
||||
# hostname-publishing L4 Service, every catalog ServiceDefinition) against what the
|
||||
# dashboard's ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides
|
||||
# half the estate and cannot tell the difference.
|
||||
#
|
||||
# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
|
||||
# service, forget the dashboard, and every cluster test stays green while the operator's
|
||||
# front door hides a running service. That is not a styling preference — it is exactly the
|
||||
# R-C2-01 failure mode, and it was a LIVE defect on main when this check was written
|
||||
# (platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root linked to it).
|
||||
#
|
||||
# THE RULE THIS IMPLEMENTS, in one sentence: every home-zone hostname the repository routes
|
||||
# over HTTP must be linked from the apex dashboard's configuration, every home-zone link the
|
||||
# apex dashboard offers must resolve to something the repository actually serves, and the two
|
||||
# R-C2-02 surfaces (health data + a per-user-authorized console) must be linked, reconciled
|
||||
# and backed by real alert rules.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the apex ingress chain, the dashboard's declared link set, every in-repo HTTPRoute,
|
||||
# every hostname-publishing L4 Service, the versioned catalog, the platform-health
|
||||
# alert rules and the console's RBAC posture are read and compared,
|
||||
# THEN — R-C2-01 —
|
||||
# (0) the INPUT SURFACE is intact: every Flux-reconciled path lies inside the roots
|
||||
# this check scans, and every manifest in them parses. A root this check does not
|
||||
# scan, or a file it cannot parse, is a service it cannot see — which would
|
||||
# silently weaken every assertion below;
|
||||
# (1) EXACTLY ONE HTTPRoute claims the bare apex (an ambiguous root is not a root),
|
||||
# and it belongs to the landing-dashboard component;
|
||||
# (2) that route pins by sectionName the ONE Gateway listener whose hostname is the
|
||||
# bare apex, and that listener is HTTPS — the wildcard `*.${BASE_DOMAIN}` listener
|
||||
# does NOT intersect the bare apex, so without the sectionName the root binds
|
||||
# nothing and serves nothing (asserted against the wildcard listener's existence,
|
||||
# so the claim is proven rather than asserted from prose);
|
||||
# (3) the apex is a SAN on the Certificate whose Secret that listener terminates with;
|
||||
# (4) the dashboard is RECONCILED, not merely authored: one Flux Kustomization points
|
||||
# at its path and is listed in the base entrypoint set;
|
||||
# (5) that Kustomization carries postBuild.substituteFrom, because the dashboard's
|
||||
# hrefs and its route's hostname are written as `${BASE_DOMAIN}` tokens — without
|
||||
# substitution the apex route binds a literal `${BASE_DOMAIN}` and EVERY tile on
|
||||
# the root links to a nonexistent host, while Flux still reports Ready;
|
||||
# (6) the component's own kustomization ships config + workload + Service + route;
|
||||
# (7) the dashboard is a service CATALOG: grouped entries carrying hrefs;
|
||||
# (8) FORWARD — every home-zone hostname an in-repo HTTPRoute serves, other than the
|
||||
# apex itself, is LINKED from the root. This is the rot-catcher;
|
||||
# (9) FORWARD — every catalog `kind: container` ServiceDefinition whose declared
|
||||
# ingress host is routed is likewise linked, tying the C2 to the VERSIONED
|
||||
# onboarding contract and not only to whatever is rendered today;
|
||||
# (10) REVERSE — every home-zone link the root offers resolves to something this repo
|
||||
# actually serves: an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service
|
||||
# (L4). A dead tile is a link the root offers to nothing;
|
||||
# (11) the off-cluster escape hatch is not a blanket one: every name declared in the
|
||||
# ConfigMap's `off-cluster-links` annotation is a specific home-zone hostname and
|
||||
# none of them shadows a hostname the repo DOES serve;
|
||||
# (12) no href names a LITERAL base-domain host. Every hostname is the `${BASE_DOMAIN}`
|
||||
# token, so prod's root cannot link dev's services — the 2026-09-04 incident class
|
||||
# (two clusters authored to describe the same names), applied to the front door;
|
||||
# — R-C2-02 —
|
||||
# (13) both health surfaces docs/12 names are linked: the Kubernetes console
|
||||
# (cluster/storage/GitOps state) and the dashboards/alerts surface;
|
||||
# (14) both are RECONCILED by Flux, so the links lead to something running;
|
||||
# (15) the health DATA those surfaces render exists as code — PrometheusRule groups for
|
||||
# each axis R-C2-02 enumerates (cluster/nodes, storage, GitOps reconciliation)
|
||||
# plus certificate expiry — and those rule files are reconciled;
|
||||
# (16) actions are CONTROLLED, not cluster-admin-in-a-browser (docs/12: "never a raw
|
||||
# cluster-admin kubeconfig in a browser"): the console chart is explicitly told NOT
|
||||
# to bind its ServiceAccount to a ClusterRole, so there is no shared privileged
|
||||
# identity to act as;
|
||||
# (17) actions are instead PER-USER: the console is configured for native OIDC with
|
||||
# credentials from a named (SOPS) Secret, not inline;
|
||||
# (18) cluster rights arrive through OIDC GROUPS at MORE THAN ONE privilege level (an
|
||||
# admin group and a lesser one) — least-privilege, not one tier for everybody;
|
||||
# (19) the C2 root DECLARES the authenticated-group posture docs/12 requires, on its own
|
||||
# route (declared, NOT enforced — see the gap note below).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE — honest gaps, the same ones the Test declaration names:
|
||||
# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; the
|
||||
# Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
|
||||
# - That the C2 is access-CONTROLLED in fact. (19) reads a DECLARED posture, because the
|
||||
# Gateway ext-authz attachment that would ENFORCE forward-auth is not built anywhere in
|
||||
# this repo (platform/homepage/httproute.yaml says so in its own header, and every
|
||||
# forward-auth app shares the gap). Declared, not enforced, and labelled as such in the
|
||||
# assertion text itself.
|
||||
# - That an operational action in the console succeeds, or that it is audited. (16)-(18)
|
||||
# prove there is no shared cluster-admin identity to act with and that rights arrive
|
||||
# per-user; the audit trail is a runtime property.
|
||||
# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
|
||||
# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
|
||||
# - That an L4-ONLY endpoint is surfaced. A raw TCP/UDP endpoint (a Minecraft game port) has
|
||||
# no URL a browser can follow, so the FORWARD direction quantifies over HTTP routes;
|
||||
# L4-published hostnames are listed as ADVISORIES and are honoured in the REVERSE
|
||||
# direction (that is how the UniFi tile, an L4 LoadBalancer on :11443, resolves).
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and the
|
||||
# requirement text above is quoted, not parsed.
|
||||
#
|
||||
# STATUS: EXECUTABLE. Pure offline: `git ls-files` + PyYAML (the dependency ten neighbouring
|
||||
# l0-static checks already take). No cluster, no network, no spec repo.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green, and rc 3 is UNWIRED for a scope-declaring test) if the landing
|
||||
# dashboard component is absent entirely — there is then no C2 root to audit, which is a
|
||||
# report about the tree, not a pass.
|
||||
if [ ! -f "${ROOT}/platform/homepage/configmap.yaml" ]; then
|
||||
echo "check SKIPPED: platform/homepage/configmap.yaml absent — no C2 landing dashboard to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: the C2 root surfaces every service the repository serves, plus health.
|
||||
|
||||
Every parse contract is documented at its parser. PyYAML rather than regex because the
|
||||
subjects are genuinely structured: multi-document apps/ bundles, and a Homepage config that
|
||||
is nested YAML *inside* a ConfigMap string value.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
ADVISORIES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if cond:
|
||||
print(f" ok {msg}")
|
||||
else:
|
||||
print(f" FAIL {msg}", file=sys.stderr)
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
# ============================================================ the hostname vocabulary
|
||||
# BASE_DOMAIN is a cluster-vars knob (R-ENV-02): every public hostname the cluster serves is
|
||||
# `<svc>.${BASE_DOMAIN}`, and dev/prod differ ONLY in that value. So a hostname in this repo
|
||||
# is canonically the TOKEN form, and the literal value below exists only to be FORBIDDEN in
|
||||
# (12): a literal is how prod came to be authored to serve dev's hostnames (2026-09-04,
|
||||
# clusters/base/cluster-vars.yaml's own header).
|
||||
TOKEN = "${BASE_DOMAIN}"
|
||||
_vars = yaml.safe_load(read(ROOT / "clusters" / "base" / "cluster-vars.yaml")) or {}
|
||||
LITERAL_BASE = ((_vars.get("data") or {}).get("BASE_DOMAIN") or "").strip()
|
||||
|
||||
|
||||
def strip_port(host: str) -> str:
|
||||
return host.split(":", 1)[0]
|
||||
|
||||
|
||||
def zone_label(host: str):
|
||||
"""The sub-label of a home-zone hostname, "" for the bare apex, or None if not in zone.
|
||||
|
||||
Accepts only the TOKEN form; a literal-base hostname returns None here and is caught
|
||||
separately by (12), so a literal can never silently satisfy a set comparison.
|
||||
"""
|
||||
h = strip_port(host)
|
||||
if h == TOKEN:
|
||||
return ""
|
||||
if h.endswith("." + TOKEN):
|
||||
return h[: -len("." + TOKEN)]
|
||||
return None
|
||||
|
||||
|
||||
def is_literal_base(host: str) -> bool:
|
||||
h = strip_port(host)
|
||||
return bool(LITERAL_BASE) and (h == LITERAL_BASE or h.endswith("." + LITERAL_BASE))
|
||||
|
||||
|
||||
# ============================================================ (0) the input surface
|
||||
# DELIVERY ROOTS — the subtrees Flux reconciles, plus catalog/ (the versioned source the
|
||||
# apps/ manifests are rendered FROM). conformance/ is deliberately EXCLUDED: its fixtures and
|
||||
# goldens declare synthetic HTTPRoutes (`paas-demo.…`, `gw-declared-conformance.…`) that are
|
||||
# test data, not services the cluster serves, and counting them would make (8) assert that
|
||||
# the operator's dashboard links this harness's fixtures.
|
||||
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
|
||||
"runbooks", "clusters", "bootstrap", "catalog")
|
||||
|
||||
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
|
||||
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
|
||||
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
|
||||
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
|
||||
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
|
||||
# routes and make (8) demand the operator's dashboard link them.
|
||||
#
|
||||
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
|
||||
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
|
||||
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
|
||||
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
|
||||
# become a real delivery root.
|
||||
NON_SERVING_RECONCILED = ("conformance/incluster",)
|
||||
|
||||
|
||||
def tracked_yaml() -> list[Path]:
|
||||
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
return [ROOT / p for p in out.stdout.split("\0") if p]
|
||||
|
||||
|
||||
ALL_TRACKED = tracked_yaml()
|
||||
DELIVERY = [p for p in ALL_TRACKED
|
||||
if p.relative_to(ROOT).parts[0] in DELIVERY_ROOTS]
|
||||
|
||||
parse_failed: list[str] = []
|
||||
DOCS: dict[Path, list[dict]] = {}
|
||||
for p in DELIVERY:
|
||||
try:
|
||||
DOCS[p] = [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
DOCS[p] = []
|
||||
parse_failed.append(str(p.relative_to(ROOT)))
|
||||
|
||||
|
||||
def docs_all():
|
||||
for p, ds in DOCS.items():
|
||||
for d in ds:
|
||||
yield p, d
|
||||
|
||||
|
||||
check(len(DELIVERY) >= 100,
|
||||
f"(0) the delivery tree is present and scanned ({len(DELIVERY)} tracked manifest(s) "
|
||||
f"under {'/, '.join(DELIVERY_ROOTS)}/)")
|
||||
check(not parse_failed,
|
||||
f"(0) every manifest in the scanned roots PARSES — an unparseable one is a service "
|
||||
f"this check cannot see, so it would weaken every set comparison below "
|
||||
f"{parse_failed if parse_failed else ''}")
|
||||
|
||||
# Every Flux-reconciled path must lie inside a root this check scans. This is the guard on
|
||||
# the guard: if a future tier reconciles, say, ./services, the forward direction would stop
|
||||
# seeing its routes and would go quietly green on a dashboard that hides all of them.
|
||||
flux_ks = [(p, d) for p, d in docs_all()
|
||||
if d.get("kind") == "Kustomization"
|
||||
and str(d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io")]
|
||||
# A RECONCILING Kustomization is one carrying a sourceRef. The pathless documents under
|
||||
# bootstrap/flux/patches/ are strategic-merge PATCH FRAGMENTS (they add a decryption stanza to
|
||||
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
|
||||
# a path of them would fail on a file that is correct.
|
||||
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
|
||||
def _reconciled_path(d):
|
||||
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
|
||||
|
||||
|
||||
unscanned = sorted({
|
||||
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
|
||||
for p, d in reconcilers
|
||||
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
|
||||
and _reconciled_path(d) not in NON_SERVING_RECONCILED
|
||||
})
|
||||
|
||||
# The exemption's own guard: an exempted tree must actually serve nothing.
|
||||
exempt_serving = sorted({
|
||||
f"{p.relative_to(ROOT)} ({d.get('kind')})"
|
||||
for p, d in docs_all()
|
||||
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
|
||||
and (d.get("kind") == "HTTPRoute"
|
||||
or (d.get("kind") == "Service"
|
||||
and any("hostname" in str(k).lower()
|
||||
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
|
||||
})
|
||||
check(not exempt_serving,
|
||||
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
|
||||
f"gains a route would hide a service from this check entirely "
|
||||
f"{exempt_serving if exempt_serving else ''}")
|
||||
check(len(reconcilers) >= 10 and not unscanned,
|
||||
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
|
||||
f"reconciled tree this check does not read is a service it cannot require a link for "
|
||||
f"{unscanned if unscanned else ''}")
|
||||
|
||||
# ============================================================ what the repo SERVES
|
||||
# L7: every hostname an in-repo HTTPRoute claims, and the component that owns it.
|
||||
routes: dict[str, list[tuple[Path, dict]]] = {}
|
||||
apex_inheriting: list[Path] = []
|
||||
for p, d in docs_all():
|
||||
if d.get("kind") != "HTTPRoute":
|
||||
continue
|
||||
hosts = (d.get("spec", {}) or {}).get("hostnames") or []
|
||||
if not hosts:
|
||||
# An HTTPRoute with NO hostnames inherits the hostname of every listener it attaches
|
||||
# to (runbooks/serving/httproute.yaml binds `https-apex` only, so it serves a PATH on
|
||||
# the apex). It introduces no new hostname, so it owes no tile.
|
||||
apex_inheriting.append(p)
|
||||
continue
|
||||
for h in hosts:
|
||||
routes.setdefault(str(h), []).append((p, d))
|
||||
|
||||
routed_labels = {} # sub-label -> owning component dir
|
||||
for h, owners in routes.items():
|
||||
lbl = zone_label(h)
|
||||
if lbl: # "" is the apex; None is out of zone
|
||||
routed_labels[lbl] = owners[0][0].parent
|
||||
|
||||
# L4: a Service that publishes a home-zone hostname to DNS is also "served by this repo" —
|
||||
# it is how UniFi (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP on :11443) reaches
|
||||
# the LAN. external-dns's annotation IS the declaration; there is no HTTPRoute to find.
|
||||
DNS_ANN = "external-dns.alpha.kubernetes.io/hostname"
|
||||
l4_labels: dict[str, Path] = {}
|
||||
for p, d in docs_all():
|
||||
if d.get("kind") != "Service":
|
||||
continue
|
||||
ann = (d.get("metadata", {}) or {}).get("annotations", {}) or {}
|
||||
for h in re.split(r"[,\s]+", str(ann.get(DNS_ANN, ""))):
|
||||
lbl = zone_label(h) if h else None
|
||||
if lbl:
|
||||
l4_labels[lbl] = p
|
||||
|
||||
check(len(routed_labels) >= 10,
|
||||
f"(0) the repo's north-south HTTP surface is discovered "
|
||||
f"({len(routed_labels)} routed home-zone hostname(s), "
|
||||
f"{len(apex_inheriting)} apex-path route(s), {len(l4_labels)} L4-published hostname(s))")
|
||||
|
||||
# ============================================================ (1) the apex is unambiguous
|
||||
apex_claims = routes.get(TOKEN, [])
|
||||
check(len(apex_claims) == 1,
|
||||
f"(1) exactly ONE HTTPRoute claims the bare apex {TOKEN} — an ambiguous root is not a "
|
||||
f"root (found {len(apex_claims)}: {[str(p.relative_to(ROOT)) for p, _ in apex_claims]})")
|
||||
apex_path, apex_route = (apex_claims[0] if apex_claims else (None, {}))
|
||||
DASH = apex_path.parent if apex_path is not None else None
|
||||
check(DASH is not None and DASH.name == "homepage",
|
||||
f"(1) the apex route belongs to the landing-dashboard component "
|
||||
f"(found {DASH.name if DASH else None!r})")
|
||||
|
||||
# ============================================================ (2) it binds the apex listener
|
||||
parents = (apex_route.get("spec", {}) or {}).get("parentRefs") or []
|
||||
section = next((pr.get("sectionName") for pr in parents if pr.get("sectionName")), None)
|
||||
gw_name = next((pr.get("name") for pr in parents), None)
|
||||
gw_ns = next((pr.get("namespace") for pr in parents), None)
|
||||
check(section is not None,
|
||||
"(2) the apex route pins a Gateway listener by sectionName — the wildcard listener's "
|
||||
"hostname cannot match a bare apex, so an unpinned apex route binds NOTHING")
|
||||
|
||||
gw = next((d for _, d in docs_all()
|
||||
if d.get("kind") == "Gateway"
|
||||
and (d.get("metadata", {}) or {}).get("name") == gw_name
|
||||
and (gw_ns is None or (d.get("metadata", {}) or {}).get("namespace") == gw_ns)), None)
|
||||
check(gw is not None, f"(2) the parent Gateway '{gw_name}' is declared in the repo")
|
||||
listeners = ((gw or {}).get("spec", {}) or {}).get("listeners") or []
|
||||
apex_listener = next((l for l in listeners if l.get("name") == section), None)
|
||||
check(apex_listener is not None,
|
||||
f"(2) the Gateway declares the '{section}' listener the apex route binds")
|
||||
check(apex_listener is not None and apex_listener.get("hostname") == TOKEN,
|
||||
f"(2) that listener's hostname is EXACTLY the bare apex "
|
||||
f"(found {(apex_listener or {}).get('hostname')!r})")
|
||||
check(apex_listener is not None and apex_listener.get("protocol") == "HTTPS",
|
||||
"(2) the apex listener terminates HTTPS (docs/12 acceptance gate: trusted TLS on the root)")
|
||||
# Prove the sectionName is LOAD-BEARING rather than decorative: a wildcard listener exists,
|
||||
# it is a DIFFERENT listener, and its hostname does not equal the apex.
|
||||
wildcard = [l for l in listeners if str(l.get("hostname", "")) == "*." + TOKEN]
|
||||
check(bool(wildcard) and all(l.get("name") != section for l in wildcard),
|
||||
f"(2) the wildcard listener(s) {[l.get('name') for l in wildcard]} are DISTINCT from "
|
||||
f"'{section}' — which is why the apex needs its own listener, and why dropping the "
|
||||
f"sectionName would leave the root unserved")
|
||||
|
||||
# ============================================================ (3) trusted TLS on the root
|
||||
cert_secret = next((ref.get("name")
|
||||
for ref in ((apex_listener or {}).get("tls", {}) or {}).get("certificateRefs") or []),
|
||||
None)
|
||||
cert = next((d for _, d in docs_all()
|
||||
if d.get("kind") == "Certificate"
|
||||
and (d.get("spec", {}) or {}).get("secretName") == cert_secret), None)
|
||||
check(cert is not None,
|
||||
f"(3) a cert-manager Certificate produces the apex listener's Secret ({cert_secret!r})")
|
||||
check(cert is not None and TOKEN in ((cert.get("spec", {}) or {}).get("dnsNames") or []),
|
||||
"(3) the bare apex is a SAN on that Certificate — the wildcard alone does NOT cover it, "
|
||||
"so without this SAN the root serves an untrusted certificate")
|
||||
|
||||
# ============================================================ (4)(5) it is RECONCILED
|
||||
dash_rel = f"./{DASH.relative_to(ROOT)}" if DASH else None
|
||||
dash_ks = [(p, d) for p, d in flux_ks
|
||||
if (d.get("spec", {}) or {}).get("path", "").rstrip("/") == dash_rel]
|
||||
check(len(dash_ks) == 1,
|
||||
f"(4) exactly one Flux Kustomization reconciles the dashboard ({dash_rel}) — authored "
|
||||
f"but unreconciled is the 'looks done, is absent' class this repo keeps hitting "
|
||||
f"(found {len(dash_ks)})")
|
||||
if dash_ks:
|
||||
base_txt = read(ROOT / "clusters" / "base" / "kustomization.yaml")
|
||||
entry = dash_ks[0][0].name
|
||||
check(re.search(rf"^\s*-\s*{re.escape(entry)}\s*(#.*)?$", base_txt, re.M) is not None,
|
||||
f"(4) that Kustomization is listed in the base entrypoint set ({entry})")
|
||||
# (5) The dashboard's hrefs and its route's hostname are ${BASE_DOMAIN} TOKENS. Without
|
||||
# postBuild.substituteFrom the token reaches the cluster LITERALLY: the apex route binds
|
||||
# the host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that
|
||||
# does not exist — and Flux still reports the Kustomization Ready. It fails OPEN.
|
||||
subs = (((dash_ks[0][1].get("spec", {}) or {}).get("postBuild") or {}).get("substituteFrom") or [])
|
||||
check(any((s.get("name") == "cluster-vars") for s in subs),
|
||||
f"(5) {dash_ks[0][0].name} carries postBuild.substituteFrom cluster-vars. It MUST: "
|
||||
f"this component's route hostname and EVERY tile href are ${{BASE_DOMAIN}} tokens, "
|
||||
f"and an unsubstituted token is applied literally while Flux still reports Ready — "
|
||||
f"a root bound to no listener and a dashboard of dead links, with a green tier")
|
||||
|
||||
# ============================================================ (6) it ships the UI
|
||||
dash_kust = yaml.safe_load(read(DASH / "kustomization.yaml")) if DASH else None
|
||||
dash_res = set((dash_kust or {}).get("resources") or [])
|
||||
for want in ("configmap.yaml", "deployment.yaml", "service.yaml", "httproute.yaml"):
|
||||
check(want in dash_res, f"(6) the dashboard kustomization ships {want}")
|
||||
|
||||
# ================================================== the dashboard's DECLARED link set
|
||||
# Homepage's config is nested YAML inside ConfigMap string values (services.yaml,
|
||||
# bookmarks.yaml). Both are read: a bookmark is a link the root offers just as a tile is.
|
||||
cm = next((d for d in (DOCS.get(DASH / "configmap.yaml") or []) if d.get("kind") == "ConfigMap"),
|
||||
{}) if DASH else {}
|
||||
cm_data = cm.get("data") or {}
|
||||
|
||||
|
||||
def embedded(key):
|
||||
try:
|
||||
return yaml.safe_load(cm_data.get(key) or "") or []
|
||||
except yaml.YAMLError:
|
||||
return None
|
||||
|
||||
|
||||
svc_cfg = embedded("services.yaml")
|
||||
bm_cfg = embedded("bookmarks.yaml")
|
||||
check(svc_cfg is not None and bm_cfg is not None,
|
||||
"(7) the dashboard's embedded services.yaml / bookmarks.yaml parse as YAML — the app "
|
||||
"mounts these verbatim, so a malformed one is a dashboard that renders nothing")
|
||||
|
||||
|
||||
def hrefs(node):
|
||||
"""Every href anywhere in Homepage's nested group/entry structure."""
|
||||
if isinstance(node, dict):
|
||||
for k, v in node.items():
|
||||
if k == "href" and isinstance(v, str):
|
||||
yield v
|
||||
else:
|
||||
yield from hrefs(v)
|
||||
elif isinstance(node, list):
|
||||
for v in node:
|
||||
yield from hrefs(v)
|
||||
|
||||
|
||||
def host_of(url):
|
||||
m = re.match(r"^https?://([^/]+)", url or "")
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
tile_urls = list(hrefs(svc_cfg or []))
|
||||
all_urls = tile_urls + list(hrefs(bm_cfg or []))
|
||||
all_hosts = [h for h in (host_of(u) for u in all_urls) if h]
|
||||
linked = {lbl for lbl in (zone_label(h) for h in all_hosts) if lbl}
|
||||
tile_labels = {lbl for lbl in (zone_label(h) for h in (host_of(u) for u in tile_urls) if h) if lbl}
|
||||
|
||||
groups = [g for g in (svc_cfg or []) if isinstance(g, dict)]
|
||||
check(len(groups) >= 1 and len(tile_labels) >= 5,
|
||||
f"(7) the root is a grouped service catalog with links "
|
||||
f"({len(groups)} group(s), {len(tile_labels)} linked home-zone service(s))")
|
||||
|
||||
# ============================================ (8) FORWARD: nothing routed is hidden
|
||||
unsurfaced = sorted(set(routed_labels) - linked)
|
||||
check(not unsurfaced,
|
||||
"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: "
|
||||
+ ", ".join(f"{n}.{TOKEN} ({routed_labels[n].relative_to(ROOT)})" for n in unsurfaced)
|
||||
if unsurfaced else
|
||||
f"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root "
|
||||
f"({len(routed_labels)} hostname(s) compared)")
|
||||
|
||||
# L4-only endpoints are reported, never required: a raw TCP/UDP port has no browsable URL.
|
||||
for lbl in sorted(set(l4_labels) - linked):
|
||||
ADVISORIES.append(
|
||||
f"{lbl}.{TOKEN} is published to DNS by an L4 Service "
|
||||
f"({l4_labels[lbl].relative_to(ROOT)}) and is not linked from the root. Not a "
|
||||
f"failure: a raw TCP/UDP endpoint has no URL a browser can follow. Verify the "
|
||||
f"service it belongs to is reachable through a tile that IS linked")
|
||||
|
||||
# ================================= (9) FORWARD: the VERSIONED onboarding contract
|
||||
cat_dir = ROOT / "catalog"
|
||||
cat_seen, cat_missing, cat_unrouted = 0, [], []
|
||||
for p in sorted(cat_dir.glob("*.yaml")):
|
||||
sd = next((d for d in (DOCS.get(p) or [])), None)
|
||||
if not isinstance(sd, dict) or sd.get("kind") != "container":
|
||||
continue # kind: vm has no north-south HTTP surface (docs/09)
|
||||
for r in sd.get("ingressRoutes") or []:
|
||||
lbl = zone_label(str(r.get("host", "")))
|
||||
if not lbl:
|
||||
continue
|
||||
if lbl not in routed_labels:
|
||||
# Declared in the catalog but not rendered into a route: an apps/ render drift
|
||||
# problem owned by R-GITOPS-02, reported here rather than silently skipped.
|
||||
cat_unrouted.append(f"{sd.get('name')}:{lbl}")
|
||||
continue
|
||||
cat_seen += 1
|
||||
if lbl not in linked:
|
||||
cat_missing.append(f"{sd.get('name')}:{lbl}")
|
||||
check(cat_seen >= 5,
|
||||
f"(9) the versioned catalog contributes routed HTTP services ({cat_seen} found) — the "
|
||||
f"C2 is tied to the onboarding contract, not only to today's rendered output")
|
||||
check(not cat_missing,
|
||||
f"(9) every catalog `kind: container` service with a route is linked from the C2 root "
|
||||
f"{cat_missing if cat_missing else ''}")
|
||||
for u in cat_unrouted:
|
||||
ADVISORIES.append(f"catalog declares an ingress host with no rendered HTTPRoute: {u} "
|
||||
f"(render drift — R-GITOPS-02 owns it, not R-C2-01)")
|
||||
|
||||
# ============================ (10)(11) REVERSE: no link points at nothing
|
||||
ann = (cm.get("metadata", {}) or {}).get("annotations", {}) or {}
|
||||
OFF_ANN = "homelab.keithjackson.dev/off-cluster-links"
|
||||
declared_off_raw = [t for t in re.split(r"[,\s]+", str(ann.get(OFF_ANN, ""))) if t]
|
||||
declared_off = {zone_label(t) for t in declared_off_raw}
|
||||
served = set(routed_labels) | set(l4_labels)
|
||||
dead = sorted(linked - served - declared_off)
|
||||
check(not dead,
|
||||
f"(10) every home-zone link the root offers resolves to something this repo serves — "
|
||||
f"an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4). Dead: "
|
||||
f"{[f'{d}.{TOKEN}' for d in dead] if dead else ''}")
|
||||
# The escape hatch is a DECLARATION, not a hole: same doctrine as an explicit `NONE` in
|
||||
# ci/touched-requirements.sh's path map. Empty today, and these two hold vacuously when it is
|
||||
# — they exist so it cannot be widened into a blanket without failing.
|
||||
check(all(zone_label(t) for t in declared_off_raw),
|
||||
f"(11) every declared off-cluster link is a specific home-zone hostname in TOKEN form "
|
||||
f"({declared_off_raw})")
|
||||
shadowed = sorted(declared_off & served)
|
||||
check(not shadowed,
|
||||
f"(11) no off-cluster declaration shadows a hostname the repo DOES serve "
|
||||
f"{shadowed if shadowed else ''}")
|
||||
|
||||
# ============================ (12) tokens, never literals
|
||||
literal_hosts = sorted({h for h in all_hosts if is_literal_base(h)})
|
||||
check(not literal_hosts,
|
||||
f"(12) no link on the root names a LITERAL '{LITERAL_BASE}' host — every hostname is the "
|
||||
f"${{BASE_DOMAIN}} token, so prod's front door cannot link dev's services (the "
|
||||
f"2026-09-04 class: two clusters authored to describe the same names). Literals: "
|
||||
f"{literal_hosts if literal_hosts else ''}")
|
||||
|
||||
# =====================================================================
|
||||
# R-C2-02 — platform health, and controlled operational actions
|
||||
# =====================================================================
|
||||
# docs/12 names the surfaces the C2 composes: Headlamp is "the R-C2-02 'controlled
|
||||
# operational actions where safe' surface"; Grafana is the dashboards/alerts surface. Each is
|
||||
# identified by the component that OWNS ITS ROUTE, not by a magic string in the config.
|
||||
CONSOLE, DASHBOARDS = "headlamp", "grafana"
|
||||
check(CONSOLE in linked,
|
||||
f"(13) the C2 root links the Kubernetes console ({CONSOLE}.{TOKEN}) — cluster, storage "
|
||||
f"and GitOps state, and the R-C2-02 operational-actions surface (docs/12)")
|
||||
check(DASHBOARDS in linked,
|
||||
f"(13) the C2 root links the dashboards/alerts surface ({DASHBOARDS}.{TOKEN})")
|
||||
|
||||
AGG = ("\n".join(read(ROOT / t / "kustomization.yaml")
|
||||
for t in ("platform", "platform-config", "platform-config-cdi")))
|
||||
for label, name in (("console", CONSOLE), ("dashboards", DASHBOARDS)):
|
||||
comp = routed_labels.get(name)
|
||||
check(comp is not None, f"(14) the {label} surface has an in-repo HTTPRoute")
|
||||
if comp is None:
|
||||
continue
|
||||
rel = f"./{comp.relative_to(ROOT)}"
|
||||
reconciled = any((d.get("spec", {}) or {}).get("path", "").rstrip("/") == rel
|
||||
for _, d in flux_ks)
|
||||
if not reconciled:
|
||||
# A component may reconcile as a MEMBER of an aggregate tier's kustomize tree
|
||||
# (platform / platform-config) instead of owning a Flux Kustomization. Both count;
|
||||
# being in NEITHER is "authored, never applied".
|
||||
reconciled = (f"\n - {comp.name}" in AGG
|
||||
or f"../{comp.relative_to(ROOT)}/" in AGG)
|
||||
check(reconciled,
|
||||
f"(14) the {label} surface ({comp.name}) is reconciled by Flux, not merely authored "
|
||||
f"— a link to an unreconciled component is a link to nothing")
|
||||
|
||||
# (15) the health DATA exists as code, for every axis R-C2-02 enumerates.
|
||||
rule_files = sorted({p for p, d in docs_all() if d.get("kind") == "PrometheusRule"})
|
||||
rule_groups = {g.get("name", "")
|
||||
for p, d in docs_all() if d.get("kind") == "PrometheusRule"
|
||||
for g in ((d.get("spec", {}) or {}).get("groups") or [])}
|
||||
for axis, needle in (("cluster/nodes", "nodes"),
|
||||
("storage", "longhorn"),
|
||||
("GitOps reconciliation", "flux"),
|
||||
("certificate expiry", "certificates")):
|
||||
check(any(needle in g for g in rule_groups),
|
||||
f"(15) platform-health alert rules exist for {axis} (a '*{needle}*' rule group) — "
|
||||
f"the C2 cannot surface health that is not measured")
|
||||
pc_txt = read(ROOT / "platform-config" / "kustomization.yaml")
|
||||
reconciled_rules = [p for p in rule_files if f"../{p.relative_to(ROOT)}" in pc_txt]
|
||||
check(len(reconciled_rules) >= 2,
|
||||
f"(15) those alert rules are RECONCILED — PrometheusRule is a CR of a CRD the platform "
|
||||
f"tier installs, so the rules are applied by platform-config "
|
||||
f"({len(reconciled_rules)}/{len(rule_files)} rule file(s) referenced there)")
|
||||
|
||||
# (16)(17) controlled actions: no shared cluster-admin identity; per-user OIDC instead.
|
||||
console_dir = routed_labels.get(CONSOLE)
|
||||
hr = next((d for p in (sorted(console_dir.glob("*.yaml")) if console_dir else [])
|
||||
for d in (DOCS.get(p) or []) if d.get("kind") == "HelmRelease"), None)
|
||||
check(hr is not None, "(16) the Kubernetes console is deployed by a HelmRelease")
|
||||
hv = ((hr or {}).get("spec", {}) or {}).get("values", {}) or {}
|
||||
check((hv.get("clusterRoleBinding") or {}).get("create") is False,
|
||||
"(16) the console chart is told NOT to bind its ServiceAccount to a ClusterRole. The "
|
||||
"chart's DEFAULT binds it to cluster-admin, which is exactly the 'raw cluster-admin "
|
||||
"kubeconfig in a browser' docs/12 forbids — every visitor would act as one shared "
|
||||
"omnipotent identity")
|
||||
oidc = ((hv.get("config") or {}).get("oidc") or {})
|
||||
check(bool(oidc),
|
||||
"(17) the console is configured for native OIDC login, so an action carries the "
|
||||
"operator's OWN identity (docs/12: Headlamp/Grafana use native OIDC)")
|
||||
check(bool((oidc.get("externalSecret") or {}).get("name")),
|
||||
"(17) its OIDC client credentials come from a NAMED (SOPS) Secret, not inline in the "
|
||||
"HelmRelease values — R-SEC-01")
|
||||
|
||||
# (18) rights arrive through GROUPS, at more than one privilege level.
|
||||
group_bindings = []
|
||||
for p, d in docs_all():
|
||||
if d.get("kind") not in ("ClusterRoleBinding", "RoleBinding"):
|
||||
continue
|
||||
for s in d.get("subjects") or []:
|
||||
if s.get("kind") == "Group" and str(s.get("name", "")).startswith("oidc:"):
|
||||
group_bindings.append(((d.get("metadata", {}) or {}).get("name"),
|
||||
s.get("name"),
|
||||
(d.get("roleRef") or {}).get("name")))
|
||||
check(len(group_bindings) >= 2,
|
||||
f"(18) cluster rights are bound to OIDC GROUPS, so access follows Keycloak group "
|
||||
f"membership rather than a per-user grant somebody must remember to revoke "
|
||||
f"({[(g, r) for _, g, r in group_bindings]})")
|
||||
check(any(r == "cluster-admin" for _, _, r in group_bindings)
|
||||
and any(r != "cluster-admin" for _, _, r in group_bindings),
|
||||
"(18) MORE THAN ONE privilege level exists (an admin group AND a lesser one) — "
|
||||
"least-privilege, not one tier for everybody (docs/12 access control, R-IDP-06)")
|
||||
|
||||
# (19) the root DECLARES the authenticated-group posture. DECLARED, NOT ENFORCED.
|
||||
route_ann = (apex_route.get("metadata", {}) or {}).get("annotations", {}) or {}
|
||||
check(route_ann.get("homelab.keithjackson.dev/auth-mode") in ("forward-auth", "oidc", "oauth2-proxy"),
|
||||
f"(19) the C2 root's route DECLARES an authentication mode "
|
||||
f"(found {route_ann.get('homelab.keithjackson.dev/auth-mode')!r}) "
|
||||
f"[DECLARED, NOT ENFORCED — no Gateway ext-authz attachment exists in this repo]")
|
||||
check(bool(route_ann.get("homelab.keithjackson.dev/auth-groups")),
|
||||
"(19) it DECLARES the authorized group(s) — docs/12 requires an authenticated user IN "
|
||||
"AN AUTHORIZED GROUP, not merely logged in "
|
||||
"[DECLARED, NOT ENFORCED — see the header's gap note]")
|
||||
|
||||
# ======================================================================== report
|
||||
print()
|
||||
print(f"C2 surface inventory (apex {TOKEN}, literal base default '{LITERAL_BASE}'):")
|
||||
print(f" routed over HTTP : {len(routed_labels)} -> {', '.join(sorted(routed_labels))}")
|
||||
print(f" L4-published : {len(l4_labels)} -> {', '.join(sorted(l4_labels)) or '(none)'}")
|
||||
print(f" linked from root : {len(linked)} -> {', '.join(sorted(linked))}")
|
||||
print(f" apex-path routes : {len(apex_inheriting)} -> "
|
||||
f"{', '.join(str(p.relative_to(ROOT)) for p in apex_inheriting) or '(none)'}")
|
||||
print(f" catalog services compared: {cat_seen} off-cluster declarations: "
|
||||
f"{sorted(declared_off_raw) or '(none)'}")
|
||||
|
||||
if ADVISORIES:
|
||||
print("\nADVISORIES (not failures):")
|
||||
for a in ADVISORIES:
|
||||
print(f" - {a}")
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: c2-root-surface — {CHECKS} assertions (R-C2-01, R-C2-02)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (c2-root-surface): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
||||
# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
|
||||
# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
|
||||
# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
|
||||
# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
|
||||
# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
|
||||
# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
|
||||
# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
|
||||
# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
|
||||
# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
||||
#
|
||||
# Three things were wrong at once, and none of them was visible from inside the cluster:
|
||||
# * the internal subnet, the Gateway VIP and a complete service inventory were publicly
|
||||
# enumerable by anyone who could guess or brute-force a label;
|
||||
# * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
|
||||
# SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
|
||||
# overrides to reach their own services;
|
||||
# * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
|
||||
# and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
|
||||
# respect.
|
||||
#
|
||||
# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
|
||||
# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
|
||||
# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
|
||||
# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
|
||||
# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
|
||||
# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
|
||||
# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
|
||||
# prose. This test makes the DNS guarantee a gate condition instead.
|
||||
#
|
||||
# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
|
||||
# Service this repo authors,
|
||||
# WHEN each catalog service's rendered routes are resolved in the applied tree (apps/),
|
||||
# THEN (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
|
||||
# annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
|
||||
# uses to skip the source object outright (source/gateway.go ->
|
||||
# annotations.IsControllerMismatch, checked before any hostname is resolved), so the
|
||||
# route cannot become an endpoint for ANY name it declares;
|
||||
# (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
|
||||
# renderer that stamped every route would satisfy (1) perfectly while silently
|
||||
# withdrawing the five public services' records, and `policy: sync` would delete them;
|
||||
# (3) the same holds for the `service` source: an internal service's rendered Service may
|
||||
# not carry an external-dns hostname annotation without the disown annotation beside it.
|
||||
# Two sources are configured, so closing only the route half is half a guarantee;
|
||||
# (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
|
||||
# the emitting function must read `exposure`, must not name any catalog service, and
|
||||
# must not be able to emit external-dns's own `dns-controller` value (the one value that
|
||||
# CLAIMS a route rather than releasing it, inverting the guarantee while looking
|
||||
# plausible). A per-service list is how twelve ServiceMonitors acquired one identical
|
||||
# wrong port name in a single edit;
|
||||
# (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
|
||||
# SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
|
||||
# annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
|
||||
# yield it (the contract's default posture is `internal`, and omitting the field is the
|
||||
# likeliest way a new service arrives). Without this the check would still pass if the
|
||||
# renderer stopped working and someone hand-maintained apps/;
|
||||
# (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
|
||||
# DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
|
||||
# scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
|
||||
# renderer to read, and they are the seven names that REMAIN publicly advertised with an
|
||||
# RFC1918 answer after this change. The register is shrink-only: an entry that stops
|
||||
# matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
|
||||
# quietly grow, and adding a new one is a review event;
|
||||
# AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
|
||||
# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
|
||||
# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
|
||||
# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
|
||||
# Same reasoning as its whole-repo siblings.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
|
||||
# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
|
||||
# and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
|
||||
# - That the record is actually GONE from Cloudflare. That is a live read against an external
|
||||
# authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
|
||||
# the positive direction (a public name DOES appear). The negative direction was proven
|
||||
# MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
|
||||
# carrying the annotation produced no record at all, and adding the annotation to a route that
|
||||
# already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
|
||||
# entry within one ~60s reconcile — but a manual proof is not a gate.
|
||||
# - That an `internal` name is unreachable from the internet. No DNS record is necessary for
|
||||
# reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
|
||||
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
|
||||
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
|
||||
# each publishes so the claim is reviewable, which is all a static check can offer.
|
||||
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
|
||||
# rather than a routable address; that is an open operator decision recorded in
|
||||
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: internal-services-publish-no-public-dns
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-02: "true"
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
|
||||
# HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
|
||||
# Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
|
||||
# ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
|
||||
# at all, so a requirement-scoped selection would skip this check on precisely the changes
|
||||
# that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
A service the operator declared `internal` cannot produce a public DNS record: the
|
||||
guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
|
||||
in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
|
||||
and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
|
||||
register so the unadjudicated set cannot grow silently.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-no-internal-service-can-publish-a-public-dns-record
|
||||
description: >-
|
||||
Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
|
||||
disown annotation on every internal one and on no public one, extend the same rule to the
|
||||
`service` source, assert catalog/render.py derives the annotation from `exposure` rather
|
||||
than from a service list, prove the shipped renderer on internal/public/default-posture
|
||||
fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+458
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
|
||||
# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
|
||||
#
|
||||
# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
|
||||
# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
|
||||
# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
|
||||
# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
|
||||
# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
|
||||
# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
|
||||
# what keeps it derived, present, and scoped.
|
||||
#
|
||||
# ASSERTIONS
|
||||
# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
|
||||
# (1) every rendered route of an `internal` service carries the disown annotation;
|
||||
# (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
|
||||
# satisfy (1) and silently delete the five public records under policy: sync;
|
||||
# (3) the `service` source half: an internal service's Service may not carry an external-dns
|
||||
# hostname annotation without the disown annotation beside it;
|
||||
# (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
|
||||
# field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
|
||||
# (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
|
||||
# public, and `exposure` OMITTED — the contract's default posture is internal);
|
||||
# (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
|
||||
# KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
|
||||
# (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
|
||||
# ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
|
||||
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
|
||||
# - That the record is gone from Cloudflare. Live read; belongs beside
|
||||
# conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
|
||||
# direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
|
||||
# annotation produced no record; adding it to a route that had one produced action=DELETE for
|
||||
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
|
||||
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
|
||||
# This check owns the NAME axis only.
|
||||
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
|
||||
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
|
||||
# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
|
||||
# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
|
||||
if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
|
||||
echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
|
||||
|
||||
PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
|
||||
line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
|
||||
structures, and a line-based reader of them is precisely the approximate parser that silently
|
||||
stops matching — the failure class this check exists to prevent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
|
||||
# ControllerKey = annotationPrefix + "controller" -> the key below
|
||||
# ControllerValue = "dns-controller" -> the value that CLAIMS an object
|
||||
# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
|
||||
# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
|
||||
CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
|
||||
CONTROLLER_CLAIMED = "dns-controller"
|
||||
HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
def docs_of(p: Path):
|
||||
"""Every YAML document in a file, tolerating the non-manifest files in the tree."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
|
||||
#
|
||||
# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
|
||||
# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
|
||||
# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
|
||||
# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
|
||||
# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
|
||||
# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
|
||||
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
|
||||
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
|
||||
#
|
||||
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
|
||||
# the honest state after this change, and it is deliberate: converting them is an operator
|
||||
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
|
||||
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
|
||||
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
|
||||
#
|
||||
# Keyed by "<relpath>:<namespace>/<name>".
|
||||
# =============================================================================================
|
||||
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
|
||||
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
|
||||
"platform component, not a catalog service; it must be reachable by every browser that "
|
||||
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
|
||||
"whether the five public services are — an operator decision, not a renderer one.",
|
||||
"platform/keycloak/httproute.yaml:keycloak/keycloak":
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
|
||||
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
|
||||
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
|
||||
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
|
||||
"is meant to be public either way; it is listed because the PUBLISHING object is the "
|
||||
"hand-authored platform route, which carries no exposure field.",
|
||||
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
|
||||
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
|
||||
"certainly should not be publicly resolvable, but it is not a catalog service and "
|
||||
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
|
||||
"internal services (Pi-hole must answer first).",
|
||||
"platform/headlamp/httproute.yaml:headlamp/headlamp":
|
||||
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
|
||||
"Keycloak OIDC; same adjudication as grafana.",
|
||||
"platform/homepage/httproute.yaml:homepage/homepage":
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
|
||||
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
|
||||
"declaration.",
|
||||
"runbooks/serving/httproute.yaml:runbooks/runbooks":
|
||||
"declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
|
||||
"contributes to the apex record rather than a name of its own (the apex TXT registry "
|
||||
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
|
||||
"header for why gating runbooks on Keycloak is a loop with no exit.",
|
||||
"platform/crafty/httproute.yaml:crafty/crafty":
|
||||
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
|
||||
"oauth2-proxy sidecar). Platform component, no exposure field.",
|
||||
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
|
||||
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
|
||||
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
|
||||
"Gateway cannot express. A game server is the one case where a public name may be "
|
||||
"genuinely wanted; the address it answers with is still RFC1918.",
|
||||
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
|
||||
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
|
||||
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
|
||||
"vm-demo, which declares no ingressRoutes and publishes nothing.",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
|
||||
# =============================================================================================
|
||||
CATALOG = ROOT / "catalog"
|
||||
sds: dict[str, dict] = {}
|
||||
for p in sorted(CATALOG.glob("*.yaml")):
|
||||
for d in docs_of(p):
|
||||
if "name" in d and "source" in d:
|
||||
sds[d["name"]] = d
|
||||
|
||||
check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
|
||||
|
||||
# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
|
||||
# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
|
||||
# renderer: a new service that omits exposure would go unaudited.
|
||||
exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
|
||||
# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
|
||||
# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
|
||||
expected_routes: dict[tuple[str, str], str] = {}
|
||||
for n, sd in sds.items():
|
||||
for r in sd.get("ingressRoutes") or []:
|
||||
expected_routes[(n, f"{n}-{r['port']}")] = n
|
||||
|
||||
internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "internal"})
|
||||
public_with_routes = sorted({s for (_, _), s in expected_routes.items()
|
||||
if exposure_of[s] == "public"})
|
||||
check(len(internal_with_routes) > 0,
|
||||
"inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
|
||||
"this check is empty, which means either the catalog changed shape or the parse is wrong")
|
||||
check(len(public_with_routes) > 0,
|
||||
"inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
|
||||
"vacuous, so the parse is presumed wrong rather than the repo")
|
||||
|
||||
SCAN_DIRS = ["apps", "platform", "runbooks"]
|
||||
routes: list[tuple[str, dict]] = [] # (relpath, HTTPRoute doc)
|
||||
dns_services: list[tuple[str, dict]] = [] # (relpath, Service doc carrying an external-dns ann)
|
||||
for sub in SCAN_DIRS:
|
||||
base = ROOT / sub
|
||||
if not base.is_dir():
|
||||
continue
|
||||
for p in sorted(base.rglob("*.yaml")):
|
||||
rel = p.relative_to(ROOT).as_posix()
|
||||
for d in docs_of(p):
|
||||
kind = d.get("kind")
|
||||
md = d.get("metadata") or {}
|
||||
ann = md.get("annotations") or {}
|
||||
if kind == "HTTPRoute":
|
||||
routes.append((rel, d))
|
||||
elif kind == "Service" and any(
|
||||
a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
|
||||
):
|
||||
dns_services.append((rel, d))
|
||||
|
||||
check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
|
||||
"contract broken (the tree is known to author several)")
|
||||
|
||||
|
||||
def ident(d: dict) -> tuple[str, str]:
|
||||
md = d.get("metadata") or {}
|
||||
return (md.get("namespace") or "", md.get("name") or "")
|
||||
|
||||
|
||||
def annotations_of(d: dict) -> dict:
|
||||
return (d.get("metadata") or {}).get("annotations") or {}
|
||||
|
||||
|
||||
def is_disowned(d: dict) -> bool:
|
||||
"""True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
|
||||
and its value is not external-dns's own. Absent == published (the default), which is the whole
|
||||
point — the annotation is the only thing standing between a route and the public zone."""
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
return v is not None and v != CONTROLLER_CLAIMED
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# (1) + (2) every internal service's rendered route is disowned; no public one is.
|
||||
# =============================================================================================
|
||||
by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
|
||||
for rel, d in routes:
|
||||
by_ident.setdefault(ident(d), []).append((rel, d))
|
||||
|
||||
for (ns, rname), svc in sorted(expected_routes.items()):
|
||||
found = by_ident.get((ns, rname), [])
|
||||
check(len(found) >= 1,
|
||||
f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
|
||||
f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
|
||||
f"(ci/render-golden.sh owns the byte-level half of this)")
|
||||
for rel, d in found:
|
||||
if exposure_of[svc] == "internal":
|
||||
v = annotations_of(d).get(CONTROLLER_KEY)
|
||||
check(
|
||||
v is not None,
|
||||
f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
|
||||
f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
|
||||
f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
|
||||
f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
|
||||
)
|
||||
check(
|
||||
v != CONTROLLER_CLAIMED,
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
|
||||
f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
|
||||
f"rather than releasing it. It reads like the guard and is its exact inverse.",
|
||||
)
|
||||
else:
|
||||
check(
|
||||
CONTROLLER_KEY not in annotations_of(d),
|
||||
f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
|
||||
f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
|
||||
f"`policy: sync` that DELETES its live public record — the five public services "
|
||||
f"must keep publishing.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (3) the `service` source half — an internal service's Service may not publish either.
|
||||
# =============================================================================================
|
||||
for rel, d in dns_services:
|
||||
ns, name = ident(d)
|
||||
svc = ns if ns in sds else None
|
||||
if svc and exposure_of[svc] == "internal":
|
||||
check(
|
||||
is_disowned(d),
|
||||
f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
|
||||
f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
|
||||
f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
|
||||
f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
|
||||
f"half is half a guarantee.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
|
||||
# =============================================================================================
|
||||
render_py = ROOT / "catalog" / "render.py"
|
||||
src = read(render_py)
|
||||
check(bool(src), "catalog/render.py unreadable")
|
||||
|
||||
emitter = re.search(
|
||||
r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
|
||||
)
|
||||
check(emitter is not None,
|
||||
"catalog/render.py must define `_externaldns_route_annotations` — the single place the "
|
||||
"ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
|
||||
"read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
|
||||
if emitter:
|
||||
body = emitter.group(0)
|
||||
# The derivation must read `exposure`. A body that does not mention it cannot be deriving
|
||||
# anything from the operator's declaration.
|
||||
check('"exposure"' in body or "'exposure'" in body,
|
||||
"`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
|
||||
"that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
|
||||
# and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
|
||||
# acquired one identical wrong port name in a single edit. Only the executable statements are
|
||||
# inspected — the docstring legitimately names the affected services as evidence.
|
||||
code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
|
||||
named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
|
||||
check(not named,
|
||||
f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
|
||||
f"annotation must be derived from `exposure` alone; a per-service list silently omits "
|
||||
f"the next service anyone adds.")
|
||||
check(CONTROLLER_CLAIMED not in code,
|
||||
f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
|
||||
f"external-dns's own value CLAIMS the object instead of releasing it")
|
||||
|
||||
# =============================================================================================
|
||||
# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
|
||||
# "internal is annotated" alone would also pass if the renderer annotated everything, and
|
||||
# because the contract's DEFAULT posture is internal — the case a new service hits by
|
||||
# omitting the field entirely, and the case a per-service list would miss.
|
||||
# =============================================================================================
|
||||
FIXTURES = [
|
||||
("fixture-internal", {"exposure": "internal"}, True,
|
||||
"an `internal` fixture must render the disown annotation"),
|
||||
("fixture-public", {"exposure": "public"}, False,
|
||||
"a `public` fixture must render NO disown annotation"),
|
||||
("fixture-default", {}, True,
|
||||
"a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
|
||||
"must also render the disown annotation"),
|
||||
]
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
for fname, extra, want, msg in FIXTURES:
|
||||
sd = {
|
||||
"name": fname,
|
||||
"kind": "container",
|
||||
"source": {"image": "example/app:1.0.0"},
|
||||
"egress": "internet",
|
||||
"dependencies": [],
|
||||
"ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
|
||||
# `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
|
||||
# keeping the fixture about DNS scoping and nothing else. A public service may not use
|
||||
# auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
|
||||
"auth": {"mode": "native"},
|
||||
"update": {"channel": "stable", "cadence": "0 4 * * 0",
|
||||
"autoRollbackOnFailedHealth": True},
|
||||
"observability": {"metrics": True, "slis": ["up"]},
|
||||
"backup": {"policy": "reacquirable"},
|
||||
}
|
||||
sd.update(extra)
|
||||
f = Path(td) / f"{fname}.yaml"
|
||||
f.write_text(yaml.safe_dump(sd, sort_keys=False))
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(render_py), str(f), "--no-image-automation"],
|
||||
capture_output=True, text=True, cwd=str(ROOT),
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
check(False, f"fixture {fname}: the shipped renderer failed: "
|
||||
f"{proc.stderr.strip()[:400]}")
|
||||
continue
|
||||
rendered = [d for d in yaml.safe_load_all(proc.stdout)
|
||||
if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
|
||||
check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
|
||||
if rendered:
|
||||
got = is_disowned(rendered[0])
|
||||
check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
|
||||
f"{annotations_of(rendered[0])})")
|
||||
|
||||
# =============================================================================================
|
||||
# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
|
||||
# declaration still names a real object.
|
||||
# =============================================================================================
|
||||
catalog_rendered = set(expected_routes)
|
||||
seen_register_keys: set[str] = set()
|
||||
for rel, d in routes + dns_services:
|
||||
ns, name = ident(d)
|
||||
if (ns, name) in catalog_rendered:
|
||||
continue
|
||||
key = f"{rel}:{ns}/{name}"
|
||||
seen_register_keys.add(key)
|
||||
check(
|
||||
key in KNOWN_NON_CATALOG_PUBLISHERS,
|
||||
f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
|
||||
f"`exposure` field this check can adjudicate, and it can put a name into the public "
|
||||
f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
|
||||
f"it is out of scope — or render it from a ServiceDefinition instead.",
|
||||
)
|
||||
stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
|
||||
check(not stale,
|
||||
f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
|
||||
f"name no object in the tree any more. Delete them — a register that keeps entries for "
|
||||
f"deleted objects stops being evidence of anything.")
|
||||
|
||||
# =============================================================================================
|
||||
# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
|
||||
# always selects this test) and NO stage is hand-written into ci/l0.sh (the
|
||||
# whole-repo-invariants-wired meta-test fails the gate if one is).
|
||||
# =============================================================================================
|
||||
TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
|
||||
test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
|
||||
check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
|
||||
check(
|
||||
re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
|
||||
is not None,
|
||||
"chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
|
||||
"mapping line — that declaration is the only thing that unions this check into every gate "
|
||||
"run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
|
||||
)
|
||||
l0_sh = read(ROOT / "ci/l0.sh")
|
||||
check(
|
||||
"internal-services-publish-no-public-dns" not in l0_sh,
|
||||
"ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
|
||||
"and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
|
||||
)
|
||||
|
||||
# =============================================================================================
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
print(f"check FAIL: {f}", file=sys.stderr)
|
||||
print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: {CHECKS} assertions passed "
|
||||
f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
|
||||
f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
|
||||
f"{len(dns_services)} external-dns-annotated Services inventoried; "
|
||||
f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
|
||||
PY
|
||||
@@ -0,0 +1,95 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
|
||||
# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
|
||||
# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
|
||||
# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
|
||||
# to nodes running a backend, and has no field that would. Combine that with
|
||||
# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
|
||||
# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
|
||||
# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
|
||||
# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
|
||||
# Service — and it never recovers on its own, because the lease holder keeps renewing after the
|
||||
# backend moves away.
|
||||
#
|
||||
# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
|
||||
# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
|
||||
# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
|
||||
# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
|
||||
# Service broken and its identical twin working is the tell. The mechanism was then confirmed
|
||||
# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
|
||||
# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
|
||||
# backend; flipped back, it vanishes again.
|
||||
#
|
||||
# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
|
||||
# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
|
||||
# running the pod" — the precise thing that is false here. The defect is invisible in review
|
||||
# and silent at apply time; it only appears as an outage, days later, after an unrelated
|
||||
# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
|
||||
# implementation; this check keeps the rest of the tree on that side of the line.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
|
||||
# that would announce its VIP,
|
||||
# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
|
||||
# node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
|
||||
# and that exemption is computed from the tree rather than hand-maintained;
|
||||
# (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
|
||||
# the check cannot pass vacuously if L2 announcement is ever removed;
|
||||
# (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
|
||||
# Cilium gains a backend-aware field the exemption logic is revisited deliberately
|
||||
# instead of going stale in silence;
|
||||
# AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
|
||||
# floor on the assertion count.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
||||
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
|
||||
# SAME script.
|
||||
#
|
||||
# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
|
||||
# that a backend exists, or that an application is listening behind it. That gap is not
|
||||
# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
|
||||
# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
|
||||
# so fixing this configuration defect did NOT by itself make those ports answer. A static
|
||||
# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
|
||||
# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
|
||||
# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
|
||||
# per-service judgement recorded in each manifest.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: l2-announced-vips-serve-from-any-node
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
|
||||
# `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
|
||||
# change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
|
||||
# under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
|
||||
# attribute NO requirement and be gated by nothing. The invariant also spans three trees at
|
||||
# once (the Services under apps/ and platform/, and the announcement policy under
|
||||
# platform-config/), which no single requirement tag can select.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
|
||||
externalTrafficPolicy Local, because the announcing node is elected without regard to
|
||||
backend placement — so an announcer holding no backend blackholes the VIP and never
|
||||
self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
|
||||
node is guaranteed a local backend.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-no-l2-announced-vip-is-etp-local
|
||||
description: >-
|
||||
Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
|
||||
would announce its VIP, and assert none combines that announcement with
|
||||
externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+242
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
|
||||
# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
|
||||
# — spec docs/00-vision-and-scope.md, section "DNS & ingress").
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
|
||||
# -----------------------------------------------------------------------------
|
||||
# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
|
||||
# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
|
||||
# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
|
||||
# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
|
||||
# actually run a backend, and it has no field that would let it.
|
||||
#
|
||||
# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
|
||||
# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
|
||||
# for the address and then has nowhere to send the packet, so the client gets ICMP port
|
||||
# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
|
||||
# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
|
||||
# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
|
||||
#
|
||||
# MEASURED on prod 2026-09-14, which is why this check exists:
|
||||
# * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
|
||||
# :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
|
||||
# * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
|
||||
# Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
|
||||
# worked. One Service broken and its twin working, from the same pod, is the whole tell.
|
||||
# * The causal mechanism was confirmed both directions on the announcing node with
|
||||
# `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
|
||||
# that node's service table entirely; flipped to `Cluster` it appears as
|
||||
# `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
|
||||
#
|
||||
# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
|
||||
# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
|
||||
# is the reference implementation (and carries the cost analysis); this check keeps the whole
|
||||
# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
|
||||
# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
|
||||
# that would announce its VIP,
|
||||
# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
|
||||
# node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
|
||||
# Service, and that exemption is computed from the tree, not hand-maintained;
|
||||
# (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
|
||||
# announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
|
||||
# this check is vacuous and must be re-derived rather than silently passing;
|
||||
# (3) the announcer's candidate set really is backend-blind — asserted structurally, by
|
||||
# requiring that the policy carries no field pinning it to backend-holding nodes;
|
||||
# AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
|
||||
# non-empty, and a floor on both the corpus and the assertion count keeps a pass
|
||||
# from degenerating into a no-op when a refactor moves these files.
|
||||
#
|
||||
# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
|
||||
# announcing node always has a local backend. Under `default-l2`'s nodeSelector
|
||||
# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
|
||||
# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
|
||||
# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
|
||||
# DaemonSet backing only.
|
||||
#
|
||||
# WHAT THIS DOES NOT PROVE (honest gaps):
|
||||
# - That the VIP is actually reachable. This is a static check; it proves the configuration
|
||||
# cannot produce the backend-blind-announcer failure, not that a backend exists, that an
|
||||
# application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
|
||||
# four Services above ALSO had no listening backend (UniFi OS was never installed in the
|
||||
# VM; no Minecraft server had been created in Crafty), and no static check can see that.
|
||||
# The end-to-end claim belongs to an L2 behavioural test against a live cluster.
|
||||
# - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
|
||||
# connection reaches the backend bearing the ingress node's address. Whether that cost is
|
||||
# acceptable is a per-service judgement recorded in each manifest, not something this
|
||||
# check can decide.
|
||||
# - Anything about Services created in-cluster rather than authored here (the Gateway's own
|
||||
# LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
|
||||
# tree, so it is outside this static scan).
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
# --------------------------------------------------------------------------- the announcers
|
||||
# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
|
||||
# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
|
||||
announcers: list[dict] = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") == "CiliumL2AnnouncementPolicy":
|
||||
spec = d.get("spec") or {}
|
||||
announcers.append({
|
||||
"file": rel,
|
||||
"name": (d.get("metadata") or {}).get("name"),
|
||||
"lb_ips": bool(spec.get("loadBalancerIPs")),
|
||||
"service_selector": spec.get("serviceSelector"),
|
||||
"spec": spec,
|
||||
})
|
||||
|
||||
check(len(announcers) > 0,
|
||||
"no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
|
||||
"LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")
|
||||
|
||||
lb_announcers = [a for a in announcers if a["lb_ips"]]
|
||||
# (2) the premise is live.
|
||||
check(len(lb_announcers) > 0,
|
||||
"no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
|
||||
"VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")
|
||||
|
||||
# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
|
||||
# nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
|
||||
# exemption logic here is revisited deliberately instead of silently going stale.
|
||||
BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
|
||||
for a in lb_announcers:
|
||||
present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
|
||||
check(not present,
|
||||
f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
|
||||
"may make announcement backend-aware — the `Local` exemption below must be re-derived "
|
||||
"before this check can keep passing (R-DNS-01)")
|
||||
|
||||
# --------------------------------------------------------------------------- DaemonSet backing
|
||||
# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
|
||||
# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
|
||||
daemonset_labels: list[tuple[str, dict]] = [] # (namespace, template labels)
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") == "DaemonSet":
|
||||
md = d.get("metadata") or {}
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
|
||||
daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))
|
||||
|
||||
|
||||
def daemonset_backed(ns, selector) -> bool:
|
||||
if not selector:
|
||||
return False
|
||||
for ds_ns, labels in daemonset_labels:
|
||||
if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- (1) the invariant
|
||||
lb_services: list[dict] = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Service":
|
||||
continue
|
||||
spec = d.get("spec") or {}
|
||||
if spec.get("type") != "LoadBalancer":
|
||||
continue
|
||||
md = d.get("metadata") or {}
|
||||
lb_services.append({
|
||||
"file": rel,
|
||||
"name": md.get("name"),
|
||||
"ns": md.get("namespace"),
|
||||
"etp": spec.get("externalTrafficPolicy"),
|
||||
"selector": spec.get("selector") or {},
|
||||
"pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
|
||||
})
|
||||
|
||||
# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
|
||||
# something moved and the check must be re-pointed rather than passing on nothing.
|
||||
check(len(lb_services) >= 3,
|
||||
f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
|
||||
"check quantifies over has moved or emptied; re-point it (R-DNS-01)")
|
||||
|
||||
exempt: list[str] = []
|
||||
for s in lb_services:
|
||||
where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
|
||||
if s["etp"] != "Local":
|
||||
CHECKS += 1 # counted: every Service is genuinely inspected
|
||||
continue
|
||||
if daemonset_backed(s["ns"], s["selector"]):
|
||||
exempt.append(f"{s['ns']}/{s['name']}")
|
||||
CHECKS += 1
|
||||
continue
|
||||
pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
|
||||
check(False,
|
||||
f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
|
||||
f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
|
||||
"announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
|
||||
"elected announcer holds no backend the VIP answers ICMP port unreachable and does "
|
||||
"not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
|
||||
"externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
|
||||
"the client-IP cost it trades away), or back the Service with a DaemonSet so every "
|
||||
"announcing node holds a backend (R-DNS-01)")
|
||||
|
||||
# --------------------------------------------------------------------------- (4) non-vacuity
|
||||
check(CHECKS >= 8,
|
||||
f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")
|
||||
|
||||
# --------------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
|
||||
print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
|
||||
f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
|
||||
f"policy(ies){note}")
|
||||
PY
|
||||
@@ -238,7 +238,7 @@ def bail(msg):
|
||||
|
||||
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
|
||||
SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
|
||||
SKIP_DIRS = {".git", "_artifacts", "node_modules", ".venv"}
|
||||
SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
|
||||
|
||||
# ------------------------------------------------------------------ discover the guard FILE
|
||||
# Hardcoding a path is what let a file that never landed disable this whole check.
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every ServiceMonitor endpoint names a scrape target that RESOLVES
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES, AND WHY ITS SIBLING COULD NOT.
|
||||
# alert-metrics-have-scrape-targets audits scrape wiring FROM THE ALERT SIDE: it walks alert rules
|
||||
# and asks whether something scrapes the metric each one reads. Its assertion (4) is the `up{job=X}`
|
||||
# axis, and it is reachable ONLY through an alert that selects that job, only where the Service and
|
||||
# the ServiceMonitor sit in the SAME FILE. Its own header names the hole. This test is the other
|
||||
# direction: it walks the MONITORS, and asks of each one whether the target it names can exist —
|
||||
# with no alert rule required anywhere.
|
||||
#
|
||||
# That difference is not academic. On 2026-09-13 all twelve apps/* ServiceMonitors named
|
||||
# `endpoints[].port: metrics` and NO app Service defined a port of that name, so each discovered
|
||||
# ZERO targets (verified on prod: `count by (job) (up)` listed 24 jobs, none of the twelve; both
|
||||
# `up{job="plex"}` and `up{job="gitea"}` returned nothing). That was caught only because each app
|
||||
# also shipped an `up{job=...}` alert. THE FIX TO THOSE ALERTS DELETED THE ONLY THING THAT MADE THE
|
||||
# BROKEN MONITORS VISIBLE — they no longer read `up{job=...}` at all — so a monitor that discovers
|
||||
# nothing would from that moment have been invisible again, and the twelve register entries that
|
||||
# recorded the gap correctly became stale and were removed. A monitor scraping nothing is a defect
|
||||
# on its own terms: it is dead configuration that reads as live coverage, and every dashboard,
|
||||
# recording rule or future alert built on its series inherits the emptiness.
|
||||
#
|
||||
# GIVEN every ServiceMonitor and PodMonitor this repo authors,
|
||||
# WHEN each endpoint's scrape target is resolved against every Service this repo authors,
|
||||
# THEN (1) every endpoint names a target AT ALL (`port` or `targetPort`) — an endpoint naming
|
||||
# neither scrapes nothing and is accepted silently by the operator;
|
||||
# (2) every endpoint that names a `port` names a port that at least one Service its selector
|
||||
# matches ACTUALLY DEFINES — the exact disagreement that made twelve monitors inert.
|
||||
# Resolution is by SELECTOR ACROSS THE WHOLE TREE, not by same-file adjacency, so moving
|
||||
# a Service into its own file cannot defeat it;
|
||||
# (3) a monitor whose selector matches no Service in this repo is UNAUDITABLE (its Service is
|
||||
# created by a Helm chart or an operator, so the port name lives outside the tree) and
|
||||
# must be DECLARED in KNOWN_EXTERNAL_MONITORS, naming who creates the Service. That
|
||||
# register is shrink-only: a monitor that later gains an in-repo Service, or that is
|
||||
# deleted, FAILS as stale, so the unauditable set cannot quietly grow;
|
||||
# (4) THE RESOLVER ITSELF IS PROVEN, on synthetic fixtures inside the check — one agreeing
|
||||
# pair it must accept, one disagreeing pair it must reject. This is not decoration: after
|
||||
# the fix there are zero in-repo monitors left for (2) to judge, and a guard whose only
|
||||
# evidence is "nothing failed" is indistinguishable from a guard that stopped working.
|
||||
# The fixtures are the exact shape catalog/render.py emits from
|
||||
# `observability.metricsEndpoint`;
|
||||
# (5) the GENERATOR cannot reintroduce the class: catalog/render.py must not hardcode an
|
||||
# endpoint port name — it must take it from the ServiceDefinition, the same declaration
|
||||
# that renders the Service port. A hardcoded literal is how twelve services acquired an
|
||||
# identical wrong port name in one edit;
|
||||
# AND (6) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
|
||||
# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps apps/* to NOTHING, so a change that
|
||||
# adds a ServiceMonitor under apps/ attributes no requirement and would filter this check out of
|
||||
# exactly the change that breaks it. Same reasoning as its sibling.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
|
||||
# the sibling check use); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (honest gaps; this is static):
|
||||
# - That a resolved target is UP, answers, or emits the series an alert wants. The sibling check
|
||||
# owns the metric-name axis and admits it cannot prove the SERIES NAME either; both are live
|
||||
# reads and belong to a future L1/L3 check against /api/v1/label/__name__/values.
|
||||
# - That a NetworkPolicy permits the scrape. The default-deny baseline drops it unless an allow is
|
||||
# rendered, which is now part of what `observability.metricsEndpoint` emits
|
||||
# (r_monitoring_ingress_policy) — but this check does not verify the allow, only the ports.
|
||||
# - For a monitor in KNOWN_EXTERNAL_MONITORS, nothing about the chart's port names. The register
|
||||
# records WHO owns them so the claim is reviewable, which is all a static check can offer.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: servicemonitor-ports-resolve
|
||||
labels:
|
||||
conformance.home.k3s/R-OBS-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-OBS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceMonitor/PodMonitor and every
|
||||
# Service in the tree (apps/wave-*, apps/vms, platform/keycloak, platform/kubevirt,
|
||||
# platform/oauth2-proxy, platform/monitoring, platform/crafty, platform/homepage,
|
||||
# runbooks/serving). No single requirement path-maps to that set, and ci/touched-requirements.sh
|
||||
# maps apps/* to no requirement at all — so a requirement-scoped selection would skip this
|
||||
# check on precisely the changes that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every ServiceMonitor/PodMonitor endpoint this repo authors names a scrape target that
|
||||
resolves — a port some matching Service actually defines — the monitors whose Services come
|
||||
from outside the tree are enumerated in a shrink-only register, and the resolver that decides
|
||||
this is itself proven on fixtures, so a monitor can never again be authored, reviewed and
|
||||
merged against a port nothing serves.
|
||||
spec:
|
||||
steps:
|
||||
- name: resolve-every-monitor-endpoint-against-every-service-this-repo-authors
|
||||
description: >-
|
||||
Inventory every ServiceMonitor/PodMonitor and Service in the tree, resolve each endpoint's
|
||||
port against the Services its selector matches, ratchet the register of monitors whose
|
||||
Service is chart-owned, prove the resolver on synthetic agreeing/disagreeing fixtures, and
|
||||
assert the renderer does not hardcode an endpoint port name.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,421 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — every ServiceMonitor/PodMonitor endpoint names a scrape target that RESOLVES.
|
||||
# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
|
||||
#
|
||||
# alert-metrics-have-scrape-targets audits scrape wiring from the ALERT side, and its `up{job=X}`
|
||||
# axis is reachable only through an alert that selects that job, only for a Service and a
|
||||
# ServiceMonitor in the SAME FILE. That is how it caught the twelve inert apps/* monitors on
|
||||
# 2026-09-13 (`endpoints[].port: metrics`, and no app Service defined such a port; on prod
|
||||
# `count by (job) (up)` listed 24 jobs and none of the twelve appeared). But FIXING those alerts
|
||||
# removed their `up{job=...}` selectors, which removed the only path by which the broken monitors
|
||||
# were visible at all. This check walks the MONITORS instead, needs no alert rule, and resolves by
|
||||
# SELECTOR across the whole tree — so a monitor that discovers nothing fails on its own terms.
|
||||
#
|
||||
# ASSERTIONS
|
||||
# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
|
||||
# (1) every endpoint names a scrape target at all — `port` or `targetPort`;
|
||||
# (2) every endpoint naming a `port` names a port at least one Service its selector matches
|
||||
# actually DEFINES;
|
||||
# (3) a monitor whose selector matches no in-repo Service is unauditable (chart/operator-owned
|
||||
# Service) and must be declared in KNOWN_EXTERNAL_MONITORS — shrink-only, stale entries FAIL;
|
||||
# (4) the RESOLVER is proven on synthetic fixtures (one agreeing pair, one disagreeing pair),
|
||||
# because after the fix there are zero in-repo monitors for (2) to judge and a guard whose
|
||||
# only evidence is "nothing failed" cannot be told apart from one that stopped working;
|
||||
# (5) catalog/render.py does not HARDCODE an endpoint port name — it derives it from the
|
||||
# ServiceDefinition, the same declaration that renders the Service port. A hardcoded literal
|
||||
# is how twelve services acquired one identical wrong port name in a single edit;
|
||||
# (6) the check is LOAD-BEARING: whole-repo scope declared here, honoured by ci/l0.sh's
|
||||
# --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written ci/l0.sh
|
||||
# stage (whole-repo-invariants-wired fails the gate if one appears).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC):
|
||||
# - Nothing about liveness: that a resolved target is UP, answers, or emits the series a rule
|
||||
# wants. The sibling check owns the metric-name axis and admits it cannot prove the series NAME
|
||||
# either. Both need a live read of /api/v1/label/__name__/values (a future L1/L3 check).
|
||||
# - Nothing about NetworkPolicy. An app namespace's default-deny baseline is `ingress: []`, so a
|
||||
# scrape is dropped unless an allow is rendered; that allow is now part of what
|
||||
# `observability.metricsEndpoint` emits (catalog/render.py r_monitoring_ingress_policy), but this
|
||||
# check verifies ports, not policy.
|
||||
# - Nothing about a chart's port names, for a monitor in KNOWN_EXTERNAL_MONITORS. The register
|
||||
# records WHO owns them so the claim is reviewable; that is all a static check can offer.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns into
|
||||
# rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the repo
|
||||
# contains no ServiceMonitor or PodMonitor at all — genuinely nothing to audit.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
|
||||
if ! grep -rlE '^[[:space:]]*kind:[[:space:]]*(Service|Pod)Monitor[[:space:]]*$' \
|
||||
"${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
|
||||
echo "check SKIPPED: no ServiceMonitor/PodMonitor found under platform/ or apps/ — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: every ServiceMonitor/PodMonitor endpoint's scrape target resolves.
|
||||
|
||||
PyYAML (the same dependency as catalog/render.py and the sibling check) rather than line-based
|
||||
parsing: a monitor's selector, namespaceSelector and endpoint list are nested structures, and a
|
||||
line-based reader of them is precisely the approximate parser that silently stops matching — the
|
||||
failure class this check exists to prevent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER — KNOWN_EXTERNAL_MONITORS. Shrink-only.
|
||||
#
|
||||
# A monitor whose selector matches NO Service in this repo cannot be audited here: the Service is
|
||||
# created by a Helm chart or an operator, so the port name lives outside the tree. That is a real
|
||||
# limit, not a pass — so each one is DECLARED, with WHO creates the Service it selects, and the
|
||||
# register can only shrink. If such a monitor later gains an in-repo Service its entry goes stale
|
||||
# and FAILS; if it is deleted, likewise. The same ratchet reasoning as the two registers in
|
||||
# alert-metrics-have-scrape-targets: failing outright would make the gate red for chart-owned
|
||||
# monitors that are perfectly correct, and a gate that is red for correct code gets disabled.
|
||||
#
|
||||
# Keyed by "<relpath>:<monitor name>".
|
||||
# =============================================================================================
|
||||
KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
|
||||
"platform/keycloak/servicemonitor.yaml:keycloak":
|
||||
"selects the operator-created `keycloak-service` (labels app=keycloak) — the Keycloak "
|
||||
"operator owns that Service. Scrapes by targetPort 9000 (the Quarkus management "
|
||||
"interface), so it names no Service port at all and assertion (2) has nothing to resolve.",
|
||||
"platform/kubevirt/servicemonitor.yaml:kubevirt":
|
||||
"selects the KubeVirt operator's own `kubevirt-prometheus-metrics` Service (label "
|
||||
"prometheus.kubevirt.io=true), created by the KubeVirt CR, not by this repo. Its `metrics` "
|
||||
"port is defined there. Live on prod: job kubevirt-prometheus-metrics, 11 targets up.",
|
||||
"platform/oauth2-proxy/servicemonitor.yaml:oauth2-proxy":
|
||||
"selects the oauth2-proxy Helm chart's Service (label app=oauth2-proxy); the chart defines "
|
||||
"its `metrics` port. Live on prod: job oauth2-proxy, 2 targets up.",
|
||||
"platform/monitoring/flux-podmonitor.yaml:flux-system":
|
||||
"a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
|
||||
"Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
|
||||
"manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
|
||||
}
|
||||
|
||||
MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
|
||||
|
||||
|
||||
def is_test_fixture(p: Path) -> bool:
|
||||
"""conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
|
||||
|
||||
The l1-rendering goldens are expected-output copies of rendered apps; the l2/l3 fixtures are
|
||||
throwaway objects a behavioural test applies itself. Auditing either would be auditing a copy —
|
||||
the same "reference, not declaration" distinction ci/touched-requirements.sh draws for fixture
|
||||
paths, and the same exclusion the sibling check makes for goldens. The templates they come from
|
||||
ARE audited, through apps/ and catalog/.
|
||||
"""
|
||||
parts = p.parts
|
||||
return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
|
||||
|
||||
|
||||
# ============================================================ (0) INVENTORY
|
||||
# services: list of (relpath, namespace, name, labels, {named ports})
|
||||
# monitors: list of (relpath, kind, namespace, name, selector, namespaceSelector, [endpoints])
|
||||
services: list[tuple] = []
|
||||
monitors: list[tuple] = []
|
||||
unparseable: list[str] = []
|
||||
|
||||
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
|
||||
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
|
||||
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
|
||||
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
|
||||
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
|
||||
continue
|
||||
txt = read(p)
|
||||
if not txt:
|
||||
continue
|
||||
if not re.search(r"^\s*kind:\s*(Service|ServiceMonitor|PodMonitor)\s*$", txt, re.M):
|
||||
continue
|
||||
rel = str(p.relative_to(ROOT))
|
||||
try:
|
||||
docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
|
||||
except yaml.YAMLError as exc:
|
||||
unparseable.append(f"{rel} ({exc})")
|
||||
continue
|
||||
for d in docs:
|
||||
kind = d.get("kind")
|
||||
md = d.get("metadata") or {}
|
||||
spec = d.get("spec") or {}
|
||||
if kind == "Service":
|
||||
names = {str(port["name"]) for port in (spec.get("ports") or [])
|
||||
if isinstance(port, dict) and port.get("name")}
|
||||
services.append((rel, md.get("namespace"), md.get("name"),
|
||||
dict(md.get("labels") or {}), names))
|
||||
elif kind in MONITOR_KINDS:
|
||||
eps = spec.get("endpoints") or spec.get("podMetricsEndpoints") or []
|
||||
monitors.append((rel, kind, md.get("namespace"), md.get("name"),
|
||||
dict((spec.get("selector") or {}).get("matchLabels") or {}),
|
||||
dict(spec.get("namespaceSelector") or {}),
|
||||
[e for e in eps if isinstance(e, dict)]))
|
||||
|
||||
for u in unparseable:
|
||||
check(False, f"{u} is not parseable YAML — every monitor and Service in it went unaudited")
|
||||
|
||||
check(len(monitors) > 0,
|
||||
"no ServiceMonitor/PodMonitor parsed out of the tree, yet the grep gate above found one. The "
|
||||
"parse contract broke and this check would pass vacuously while every scrape target went "
|
||||
"unaudited")
|
||||
check(len(services) >= 10,
|
||||
f"only {len(services)} Service(s) parsed from the tree. This repo authors more than that "
|
||||
f"(every apps/wave-* ServiceDefinition renders one), so the inventory has stopped being "
|
||||
f"populated — assertion (2) would then resolve nothing and pass for the wrong reason")
|
||||
|
||||
|
||||
# ============================================================ THE RESOLVER
|
||||
def monitor_namespaces(mon_ns, ns_selector) -> set | None:
|
||||
"""Namespaces a monitor selects. None means ANY (`any: true`).
|
||||
|
||||
An empty namespaceSelector defaults to the monitor's own namespace — the operator's documented
|
||||
behaviour, and the case every repo-authored monitor uses.
|
||||
"""
|
||||
if ns_selector.get("any") is True:
|
||||
return None
|
||||
names = ns_selector.get("matchNames")
|
||||
if names:
|
||||
return {str(n) for n in names}
|
||||
return {mon_ns} if mon_ns else set()
|
||||
|
||||
|
||||
def matching_services(selector: dict, namespaces: set | None) -> list[tuple]:
|
||||
"""Services whose labels are a SUPERSET of the selector and whose namespace is in range.
|
||||
|
||||
An EMPTY selector matches every Service in range (Kubernetes label-selector semantics). A
|
||||
PodMonitor selector is compared against Service labels too — deliberately, so the resolver has
|
||||
one code path; when that yields no match the monitor lands in the unauditable register with the
|
||||
reason recorded, which is the honest outcome for a pod-side target this repo does not author.
|
||||
"""
|
||||
out = []
|
||||
for svc in services:
|
||||
_, svc_ns, _, labels, _ = svc
|
||||
if namespaces is not None and svc_ns not in namespaces:
|
||||
continue
|
||||
if all(labels.get(k) == v for k, v in selector.items()):
|
||||
out.append(svc)
|
||||
return out
|
||||
|
||||
|
||||
def resolve(mon: tuple) -> tuple[list[str], list[tuple]]:
|
||||
"""(failures, matched services) for one monitor. The unit assertions (1) and (2) live here so
|
||||
the SELF-TEST at the bottom exercises the same code path the repo audit does."""
|
||||
rel, kind, ns, name, selector, ns_sel, eps = mon
|
||||
site = f"{rel}:{name}"
|
||||
problems: list[str] = []
|
||||
matched = matching_services(selector, monitor_namespaces(ns, ns_sel))
|
||||
|
||||
for i, ep in enumerate(eps):
|
||||
if not ep.get("port") and not ep.get("targetPort"):
|
||||
problems.append(
|
||||
f"{site} endpoint[{i}] names NEITHER `port` NOR `targetPort`. The operator accepts "
|
||||
f"that silently and the monitor scrapes nothing")
|
||||
if not eps:
|
||||
problems.append(
|
||||
f"{site} has NO endpoints at all — it can never produce a target, and nothing about "
|
||||
f"the object looks wrong")
|
||||
|
||||
named = [(i, str(ep["port"])) for i, ep in enumerate(eps) if ep.get("port")]
|
||||
if named and matched:
|
||||
defined: set = set()
|
||||
for svc in matched:
|
||||
defined |= svc[4]
|
||||
for i, port in named:
|
||||
if port not in defined:
|
||||
where = ", ".join(f"{s[0]}:{s[2]}" for s in matched)
|
||||
problems.append(
|
||||
f"{site} endpoint[{i}] scrapes port {port!r}, which NO Service its selector "
|
||||
f"matches defines. Matched Service(s): {where}; the port names they define are "
|
||||
f"{sorted(defined) or ['(none)']}. The monitor therefore discovers ZERO "
|
||||
f"targets: no `up{{job=\"{name}\"}}` series exists, anything built on its "
|
||||
f"series is empty, and PrometheusTargetsMissing cannot see it either because a "
|
||||
f"target that was never discovered has no up series to be 0. Give the Service a "
|
||||
f"named port the workload actually serves metrics on (for a ServiceDefinition: "
|
||||
f"`observability.metricsEndpoint`, which renders the Service port, this "
|
||||
f"endpoint and the monitoring ingress allow together), or scrape by targetPort")
|
||||
return problems, matched
|
||||
|
||||
|
||||
# ============================================================ (1)(2)(3) THE REPO AUDIT
|
||||
auditable = 0
|
||||
for mon in monitors:
|
||||
rel, kind, ns, name, selector, ns_sel, eps = mon
|
||||
site = f"{rel}:{name}"
|
||||
problems, matched = resolve(mon)
|
||||
for msg in problems:
|
||||
check(False, msg)
|
||||
if not problems:
|
||||
# An endpoint-shape assertion passed for this monitor; count it so the report is honest
|
||||
# about how much of the run was real work.
|
||||
check(True, f"{site}: every endpoint names a scrape target")
|
||||
if matched:
|
||||
auditable += 1
|
||||
check(site not in KNOWN_EXTERNAL_MONITORS,
|
||||
f"{site} is listed in KNOWN_EXTERNAL_MONITORS but its selector NOW matches "
|
||||
f"{len(matched)} Service(s) this repo authors — the register is shrink-only: delete "
|
||||
f"the entry so the monitor is audited for real")
|
||||
else:
|
||||
check(site in KNOWN_EXTERNAL_MONITORS,
|
||||
f"{site} selects {selector or '(everything)'} in namespace(s) "
|
||||
f"{sorted(monitor_namespaces(ns, ns_sel) or ['<any>'])} and NO Service in this repo "
|
||||
f"matches, so its endpoint ports cannot be audited here. Either the selector is wrong "
|
||||
f"(a monitor matching nothing discovers nothing), or the Service is chart/operator "
|
||||
f"owned — in which case DECLARE it in KNOWN_EXTERNAL_MONITORS with who creates it")
|
||||
|
||||
for site in sorted(KNOWN_EXTERNAL_MONITORS):
|
||||
check(any(f"{m[0]}:{m[3]}" == site for m in monitors),
|
||||
f"KNOWN_EXTERNAL_MONITORS declares {site}, which is not a monitor in this repo any more "
|
||||
f"— a stale exemption hides the next real gap; delete the entry")
|
||||
|
||||
|
||||
# ============================================================ (4) THE RESOLVER IS PROVEN
|
||||
# After the twelve apps/* monitors were removed there is NO in-repo monitor left for assertion (2)
|
||||
# to judge (every survivor is chart-owned, and keycloak's scrapes by targetPort). A guard in that
|
||||
# state passes whether it works or not. So the resolver is run against two synthetic monitors over
|
||||
# two synthetic Services, in the exact shape catalog/render.py emits from
|
||||
# `observability.metricsEndpoint`, and BOTH outcomes are asserted.
|
||||
_real_services = services
|
||||
services = [
|
||||
("<selftest>", "fixture", "agreeing", {"app.kubernetes.io/name": "agreeing"},
|
||||
{"port-8080", "metrics"}),
|
||||
("<selftest>", "fixture", "disagreeing", {"app.kubernetes.io/name": "disagreeing"},
|
||||
{"port-8080"}),
|
||||
]
|
||||
good = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
|
||||
{"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
|
||||
[{"port": "metrics", "path": "/metrics", "interval": "30s"}])
|
||||
bad = ("<selftest>", "ServiceMonitor", "fixture", "disagreeing",
|
||||
{"app.kubernetes.io/name": "disagreeing"}, {"matchNames": ["fixture"]},
|
||||
[{"port": "metrics", "path": "/metrics", "interval": "30s"}])
|
||||
empty = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
|
||||
{"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
|
||||
[{"path": "/metrics"}])
|
||||
|
||||
good_problems, good_matched = resolve(good)
|
||||
bad_problems, bad_matched = resolve(bad)
|
||||
empty_problems, _ = resolve(empty)
|
||||
services = _real_services
|
||||
|
||||
check(good_matched and not good_problems,
|
||||
f"SELF-TEST: the resolver rejected a CORRECT monitor (Service defines `metrics`, endpoint "
|
||||
f"scrapes `metrics`): {good_problems or 'selector matched no fixture Service'}. The resolver "
|
||||
f"is broken in the direction that makes the gate red for correct code")
|
||||
check(bad_matched and len(bad_problems) == 1 and "discovers ZERO targets" in bad_problems[0],
|
||||
"SELF-TEST: the resolver ACCEPTED a monitor whose endpoint names a port its Service does not "
|
||||
"define — the exact defect that left twelve apps/* monitors discovering nothing on prod. "
|
||||
"Assertion (2) is inert; every pass above proves nothing")
|
||||
check(len(empty_problems) == 1 and "NEITHER" in empty_problems[0],
|
||||
"SELF-TEST: the resolver ACCEPTED an endpoint naming neither `port` nor `targetPort`, so "
|
||||
"assertion (1) is inert")
|
||||
|
||||
|
||||
# ============================================================ (5) THE GENERATOR CANNOT REGRESS
|
||||
RENDER = read(ROOT / "catalog" / "render.py")
|
||||
check(RENDER != "", "catalog/render.py is unreadable — the generator half cannot be audited")
|
||||
sm_fn = RENDER.split("def r_servicemonitor")[-1].split("\ndef ")[0] if RENDER else ""
|
||||
check("def r_servicemonitor" in RENDER,
|
||||
"catalog/render.py no longer defines r_servicemonitor — this assertion is auditing nothing")
|
||||
check(not re.search(r'"port":\s*"[a-z][a-z0-9-]*"', sm_fn),
|
||||
"catalog/render.py's r_servicemonitor HARDCODES a literal endpoint port name. That is the "
|
||||
"original defect's mechanism: one edit gave twelve services the same wrong port name "
|
||||
"(`metrics`), which no rendered Service defined, and every `up`-based SLI built on them was "
|
||||
"structurally inert on prod. The port name must come from the ServiceDefinition's "
|
||||
"`observability.metricsEndpoint`, the same declaration that renders the Service port")
|
||||
check("metricsEndpoint" in sm_fn,
|
||||
"catalog/render.py's r_servicemonitor no longer reads `metricsEndpoint`, so it is back to "
|
||||
"rendering a monitor from an intent (`metrics: true`) that cannot say where metrics are "
|
||||
"served — for most of this catalog, nowhere")
|
||||
SCHEMA = read(ROOT / "schemas" / "servicedefinition.schema.json")
|
||||
check("metricsEndpoint" in SCHEMA,
|
||||
"schemas/servicedefinition.schema.json does not define observability.metricsEndpoint, so the "
|
||||
"field the renderer depends on is not part of the contract and `additionalProperties: false` "
|
||||
"would reject any ServiceDefinition that declares one")
|
||||
|
||||
|
||||
# ============================================================ (6) LOAD-BEARING
|
||||
SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "servicemonitor-ports-resolve"
|
||||
/ "chainsaw-test.yaml")
|
||||
check(re.search(r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo", SELF, re.M) is not None,
|
||||
"servicemonitor-ports-resolve/chainsaw-test.yaml does not declare "
|
||||
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
|
||||
"change happens to touch R-OBS-01. ci/touched-requirements.sh maps apps/* to NOTHING, so "
|
||||
"adding a ServiceMonitor under apps/ would be filtered out of exactly the change that "
|
||||
"reintroduces the defect")
|
||||
L0 = read(ROOT / "ci" / "l0.sh")
|
||||
GATE = read(ROOT / "ci" / "gate.sh")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
|
||||
"purely requirement-scoped run again and the scope annotation above is inert decoration")
|
||||
check("servicemonitor-ports-resolve" not in L0,
|
||||
"ci/l0.sh names this check in a hand-written stage. Whole-repo invariants are selected BY "
|
||||
"DECLARING the scope; a per-test stage is the pattern tests/l0-static/"
|
||||
"whole-repo-invariants-wired fails the gate for, because the next invariant's author has to "
|
||||
"REMEMBER to add one and the omission is silent")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
|
||||
and re.search(r"fail\).*BLOCK=1", GATE) is not None,
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
|
||||
# ======================================================================== report
|
||||
print(f"Monitor inventory: {len(monitors)} ServiceMonitor/PodMonitor(s), {len(services)} "
|
||||
f"Service(s) authored in this repo")
|
||||
for rel, kind, ns, name, selector, ns_sel, eps in monitors:
|
||||
tgt = ", ".join(
|
||||
str(e.get("port") or f"targetPort {e.get('targetPort')}" or "<none>") for e in eps
|
||||
) or "<no endpoints>"
|
||||
mark = "EXTERNAL" if f"{rel}:{name}" in KNOWN_EXTERNAL_MONITORS else "resolved"
|
||||
print(f" {mark:<8} {kind:<14} {rel}:{name} -> {tgt}")
|
||||
print(f"\n resolvable against an in-repo Service: {auditable}/{len(monitors)}")
|
||||
print(f" DECLARED GAP (shrink-only): KNOWN_EXTERNAL_MONITORS: {len(KNOWN_EXTERNAL_MONITORS)} "
|
||||
f"monitor(s) whose Service is created by a chart/operator outside this tree")
|
||||
print(" RESOLVER SELF-TEST: agreeing pair accepted, disagreeing pair rejected, "
|
||||
"target-less endpoint rejected")
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: servicemonitor-ports-resolve — {CHECKS} assertions (R-OBS-01)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (servicemonitor-ports-resolve): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
|
||||
fi
|
||||
|
||||
grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH"
|
||||
chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
"conformance/lib/common.sh no longer resolves the scope annotation — no test can be
|
||||
selected BY declaring a scope, so every whole-repo invariant is back to being
|
||||
selectable away by the requirement filter"
|
||||
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
|
||||
chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
|
||||
chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
"conformance/run.sh does not union scope-declared tests (--include-scope /
|
||||
conf_dirs_for_scopes missing) — the annotation would be inert decoration"
|
||||
|
||||
grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
|
||||
chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
"ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
|
||||
purely requirement-scoped run, and every whole-repo invariant is filtered out of
|
||||
exactly the change that breaks it"
|
||||
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
|
||||
CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
|
||||
)
|
||||
MIN_DECLARED="${MIN_DECLARED:-5}"
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
|
||||
chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
|
||||
chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
"only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
|
||||
annotation was stripped from the whole-repo checks, or the resolver stopped matching
|
||||
it. Every assertion below would then pass while checking nothing"
|
||||
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
rel="${d#"$ROOT"/}"
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
|
||||
chk $? "${rel}: declares the scope AND carries an executable offline check" \
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
|
||||
chk $arc "${rel}: declares the scope AND carries an executable offline check" \
|
||||
"${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
|
||||
unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
|
||||
permanent not-green, not a guard"
|
||||
@@ -85,31 +85,64 @@ done
|
||||
# second would fail.
|
||||
# Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves
|
||||
# identically — but the literal here does not match ci/touched-requirements.sh's content-scan
|
||||
# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
|
||||
# pattern `R-[A-Z0-9]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
|
||||
# to the change. (Same reason the synthetic fixture below carries no requirement label at all.)
|
||||
BOGUS_REQ="r-nosuchreq-99"
|
||||
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
|
||||
sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
|
||||
|
||||
[ -z "$sel_without" ]
|
||||
chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
[ -z "$sel_without" ] && arc=0 || arc=$?
|
||||
chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
"the control case is broken: --requirements ${BOGUS_REQ} still selected tests
|
||||
($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
|
||||
about --include-scope"
|
||||
|
||||
# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
|
||||
# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
|
||||
# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
|
||||
# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
|
||||
# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
|
||||
# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
|
||||
missing=""
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
|
||||
printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ -z "$missing" ]
|
||||
chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
[ -z "$missing" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
"the scope union does not actually select:${missing}. This is the blind spot itself: a
|
||||
change that attributes no requirement would not run these checks"
|
||||
|
||||
# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
|
||||
# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
|
||||
# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
|
||||
# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
|
||||
# three rendering goldens stale.
|
||||
sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
|
||||
--print-selection 2>/dev/null || true)"
|
||||
missing_l1=""
|
||||
declared_l1=0
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
|
||||
declared_l1=$((declared_l1+1))
|
||||
printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
|
||||
"scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
|
||||
${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
|
||||
when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
|
||||
|
||||
# And the L1 union must not leak the other way either.
|
||||
strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
|
||||
[ -z "$strays_l1" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
|
||||
"--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
|
||||
|
||||
# The union must not smuggle in another layer's tests when --layer bounds the run.
|
||||
strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
|
||||
[ -z "$strays" ]
|
||||
chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
[ -z "$strays" ] && arc=0 || arc=$?
|
||||
chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
"--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
|
||||
|
||||
# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
|
||||
@@ -143,23 +176,23 @@ run_offline() { # run_offline <dir...> -> echoes rc
|
||||
rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
|
||||
rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
|
||||
|
||||
[ "$rc_declared" = 3 ]
|
||||
chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
[ "$rc_declared" = 3 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
"a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
|
||||
whole-repo invariant reporting 'nothing to audit' means the tree it audits has
|
||||
VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
|
||||
rc=3. Exit 0 here turns that back into a silent green"
|
||||
|
||||
[ "$rc_undeclared" = 0 ]
|
||||
chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
|
||||
chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
"a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
|
||||
rule leaked onto ordinary scaffolds, which would make every unfinished test block"
|
||||
|
||||
# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
|
||||
l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
|
||||
printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
|
||||
chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
|
||||
chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
"ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
|
||||
check that SKIPPED would be reported as a PASS — the exact false green the deleted
|
||||
stages were written to prevent"
|
||||
@@ -193,8 +226,8 @@ done
|
||||
|
||||
# ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
|
||||
# Carried over from the three deleted stages' own checks — asserted once, here.
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
|
||||
chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
|
||||
chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
"ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
|
||||
would hold while a red L0 stage merged anyway"
|
||||
|
||||
|
||||
@@ -23,6 +23,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-PLAT-02"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
|
||||
spec:
|
||||
|
||||
@@ -196,6 +196,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -212,26 +213,6 @@ spec:
|
||||
- name: paas-demo
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: paas-demo
|
||||
namespace: paas-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: paas-demo
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: paas-demo
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- paas-demo
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
kind: ImageRepository
|
||||
|
||||
@@ -27,6 +27,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
Rendering any service's namespace yields a default-deny baseline policy.
|
||||
spec:
|
||||
|
||||
@@ -72,26 +72,6 @@ spec:
|
||||
- toEntities:
|
||||
- world
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: baseline-demo
|
||||
namespace: baseline-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: baseline-demo
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: baseline-demo
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- baseline-demo
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
kind: ImageRepository
|
||||
|
||||
@@ -32,6 +32,17 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
||||
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
||||
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
||||
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
||||
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
||||
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
||||
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
||||
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
||||
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
||||
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/intent: >-
|
||||
egress: vpn renders the egress-gateway + kill-switch path; internet does not.
|
||||
spec:
|
||||
|
||||
+3
-22
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -239,26 +240,6 @@ spec:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: sonarr
|
||||
namespace: sonarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- sonarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -273,14 +254,14 @@ spec:
|
||||
- name: sonarr.slis
|
||||
rules:
|
||||
- alert: sonarr-down
|
||||
expr: up{job="sonarr"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
annotations:
|
||||
summary: sonarr is down
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
|
||||
@@ -223,6 +223,7 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
@@ -239,26 +240,6 @@ spec:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: sonarr
|
||||
namespace: sonarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- sonarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -273,14 +254,14 @@ spec:
|
||||
- name: sonarr.slis
|
||||
rules:
|
||||
- alert: sonarr-down
|
||||
expr: up{job="sonarr"} == 0
|
||||
expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
annotations:
|
||||
summary: sonarr is down
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
|
||||
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
|
||||
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
|
||||
|
||||
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
|
||||
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
|
||||
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
|
||||
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
|
||||
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
|
||||
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
|
||||
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
|
||||
> 2026-09-29, lie about it.
|
||||
>
|
||||
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
|
||||
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
|
||||
> not enable L2/L3, it only changes which way this stage fails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
# infra/gitea — platform settings that are NOT GitOps-managed
|
||||
|
||||
Gitea's own repo settings (branch protection, required status checks, approvals) live
|
||||
in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
|
||||
and invisible to the gate** — exactly how the state below went unnoticed.
|
||||
|
||||
## The gap this closes
|
||||
|
||||
Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
|
||||
(2026-08-31) asserting the opposite:
|
||||
|
||||
| Claimed | Measured |
|
||||
|---|---|
|
||||
| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
|
||||
| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
|
||||
|
||||
So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
|
||||
rule; this directory is how the rule gets enforced by the machine.
|
||||
|
||||
## `branch-protection.sh`
|
||||
|
||||
Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
|
||||
do not trust the API response**, since Gitea may let site admins bypass protection.
|
||||
|
||||
./infra/gitea/branch-protection.sh
|
||||
|
||||
It authenticates with the credential `git` already has stored for this host, and never
|
||||
prints the secret.
|
||||
|
||||
## `runner-token-rotate.sh`
|
||||
|
||||
Mints a fresh instance-level registration token from the Gitea API and writes it into
|
||||
`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
|
||||
never printed.**
|
||||
|
||||
./infra/gitea/runner-token-rotate.sh
|
||||
|
||||
A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
|
||||
and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
|
||||
longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
|
||||
and polls with an identity Gitea does not recognise — so it never appears in the runner
|
||||
list. That is precisely the observed symptom: zero runners, pod running, no error anyone
|
||||
looks at. Clearing the PVC is what forces a real re-registration; the script prints the
|
||||
commands.
|
||||
|
||||
## Sequence — DONE 2026-09-29
|
||||
|
||||
All four steps are complete and each was verified, not assumed:
|
||||
|
||||
| Step | State |
|
||||
|---|---|
|
||||
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
||||
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
||||
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
||||
|
||||
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
||||
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
||||
account agents authenticate as.
|
||||
|
||||
### 5. Block the admin override — the step the first pass MISSED
|
||||
|
||||
`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
|
||||
bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
|
||||
merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
|
||||
like the gate were overridable by precisely the account that merges.
|
||||
|
||||
**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
|
||||
`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
|
||||
*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
|
||||
(`/branch_protections/main`) and enumerate its keys.
|
||||
|
||||
> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
|
||||
> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
|
||||
> means "key absent". Enumerate with `to_entries` and act on what is really there.
|
||||
|
||||
Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
|
||||
proven when a green PR is REFUSED a merge until approved, and not before.
|
||||
|
||||
### `require-gate.sh` / `verify-then-enforce.sh`
|
||||
|
||||
`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
|
||||
merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
|
||||
the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
|
||||
enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
|
||||
|
||||
Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
|
||||
|
||||
### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
|
||||
|
||||
`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
|
||||
a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
|
||||
at least one runner is online. It is the only thing in this repository that can see Gitea's own
|
||||
database.
|
||||
|
||||
It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
|
||||
`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
|
||||
|
||||
Behaviour, proven by test rather than assumed:
|
||||
|
||||
| Situation | Exit | Effect |
|
||||
|---|---|---|
|
||||
| live matches the declaration | 0 | green |
|
||||
| any key drifted, or no runner online | 1 | **blocks the merge** |
|
||||
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
|
||||
|
||||
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
|
||||
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
|
||||
the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
|
||||
`gate.yml` so a future missing token blocks instead of skipping.
|
||||
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
### Why these scripts exist at all, and what is still missing
|
||||
|
||||
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
|
||||
conformance gate can read. Nothing detects drift from what this directory declares. That is
|
||||
the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
|
||||
gone for a month while the log recorded both as working, and **no check existed that could
|
||||
have caught it.**
|
||||
|
||||
These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
|
||||
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
|
||||
root cause of the 2026-09-29 session.
|
||||
|
||||
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
|
||||
detector detects nothing. That is the remaining step.
|
||||
|
||||
## Historical sequence (kept for context)
|
||||
|
||||
### Original ordering notes
|
||||
|
||||
1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
|
||||
the runner actually re-registers, then confirm it shows `online` in the API. Do not
|
||||
skip the confirmation: the whole reason this went unnoticed is that a runner was
|
||||
*recorded* as online without anyone checking.
|
||||
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
||||
3. **Only then set `enable_status_check: true`** with that exact context.
|
||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||
|
||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||
> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
|
||||
> reported.
|
||||
|
||||
### Measured, once the runner came up (2026-09-29)
|
||||
|
||||
The first CI jobs this repo ever ran, on PR #1 and PR #2:
|
||||
|
||||
| Context | Result | Time |
|
||||
|---|---|---|
|
||||
| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
|
||||
| `gate / gate (push)` | success | 1m32s (PR #2 only) |
|
||||
| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
|
||||
|
||||
**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
|
||||
but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
|
||||
does not hang a status check: Gitea reports it as **`success`** with the description
|
||||
"Has been skipped". So requiring it would not deadlock anything — it would pass
|
||||
*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
|
||||
a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
|
||||
exactly the fail-open shape this repo keeps finding.
|
||||
|
||||
The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
|
||||
(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
|
||||
is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
|
||||
`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
|
||||
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
|
||||
"_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
|
||||
"sensitive_paths": [
|
||||
"ci/**",
|
||||
".gitea/**",
|
||||
"conformance/**",
|
||||
"clusters/**",
|
||||
"bootstrap/**",
|
||||
"infra/gitea/**",
|
||||
"platform/kyverno/**",
|
||||
"platform-config/kustomization.yaml",
|
||||
".sops.yaml",
|
||||
"**/*.sops.yaml",
|
||||
"**/rbac.yaml",
|
||||
"CLAUDE.md"
|
||||
],
|
||||
"_why_each": {
|
||||
"ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
|
||||
"conformance/**": "the tests and the harness; changing them changes what green MEANS",
|
||||
"clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
|
||||
"infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
|
||||
"platform/kyverno/**": "the admission policy engine",
|
||||
".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
|
||||
"CLAUDE.md": "the delivery contract agents follow"
|
||||
}
|
||||
}
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stop repository admins from overriding the merge requirements on `main`.
|
||||
#
|
||||
# WHY. Gitea lets repo admins override branch-protection merge requirements unless
|
||||
# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
|
||||
# required_approvals and the required status check were both overridable by the one account
|
||||
# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
|
||||
# `required_approvals: 1`.
|
||||
#
|
||||
# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
|
||||
# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
|
||||
# real keys (`to_entries`) rather than probing names you assume exist.
|
||||
#
|
||||
# Recoverable: an owner can PATCH this back to false if it gets in the way.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
|
||||
|
||||
echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
|
||||
api -X PATCH -H 'Content-Type: application/json' \
|
||||
-d '{"block_admin_merge_override": true}' "$B" >/dev/null
|
||||
AFTER=$(api "$B" | jq -r .block_admin_merge_override)
|
||||
echo "==> after: block_admin_merge_override = $AFTER"
|
||||
[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
|
||||
|
||||
api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
|
||||
cat <<'NOTE'
|
||||
|
||||
NOW PROVE IT -- the setting is not the evidence. On the next PR:
|
||||
* let the gate go green, then try to merge WITHOUT approving it.
|
||||
* Gitea must REFUSE. If it merges, this still is not doing its job.
|
||||
Do not record approvals as enforced until you have seen a merge refused.
|
||||
NOTE
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# Enable branch protection on main for home-network-k3s-impl.
|
||||
#
|
||||
# Sets enable_push:false -> no direct pushes to main; changes land only by merging a PR.
|
||||
#
|
||||
# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
|
||||
# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
|
||||
# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
|
||||
# Turn it on only after a runner is live and you have seen the gate actually report on
|
||||
# a real PR, using the context name CI itself produced.
|
||||
#
|
||||
# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
|
||||
# a repo collaborator, so requiring an approval now could leave nobody able to approve.
|
||||
# Add the collaborator first, then raise this to 1.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev
|
||||
OWNER=Home-Network
|
||||
REPO=home-network-k3s-impl
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"
|
||||
P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
|
||||
echo "==> creating branch protection on 'main'"
|
||||
jq -n '{
|
||||
branch_name: "main",
|
||||
enable_push: false,
|
||||
enable_status_check: false,
|
||||
status_check_contexts: [],
|
||||
required_approvals: 0,
|
||||
block_on_rejected_reviews: true,
|
||||
block_on_outdated_branch: false,
|
||||
dismiss_stale_approvals: true
|
||||
}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
-X POST -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
|
||||
| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
|
||||
|
||||
echo "==> verifying (this is the part that matters)"
|
||||
curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Now prove it actually blocks you -- do not take the API's word for it. From a clean
|
||||
checkout on main, with nothing to lose:
|
||||
|
||||
git commit --allow-empty -m 'protection probe'
|
||||
git push origin main # EXPECT: rejected by the remote
|
||||
git reset --hard HEAD~1 # tidy up either way
|
||||
|
||||
If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
|
||||
yet doing what you want.
|
||||
NOTE
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
"enable_push": false,
|
||||
"enable_push_whitelist": false,
|
||||
"enable_merge_whitelist": false,
|
||||
"enable_status_check": true,
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
"dismiss_stale_approvals": true
|
||||
}
|
||||
}
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
|
||||
#
|
||||
# RUN THIS LAST. Prerequisites, in order:
|
||||
# a) PR #7 (dind native sidecar) is MERGED, and
|
||||
# b) you have SEEN a real workflow run go green on the converted pod.
|
||||
# The runner Deployment is the merge gate for the whole repo. Turning on a required check
|
||||
# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
|
||||
# an owner can set enable_status_check back to false. But do not rely on that.)
|
||||
#
|
||||
# Requires exactly ONE context: `gate / gate (pull_request)`.
|
||||
# * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
|
||||
# SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
|
||||
# ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
|
||||
# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
|
||||
# lets repo admins override merge requirements without it, so required_approvals and the
|
||||
# required status check are both bypassable by exactly the account that does the merging.
|
||||
# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
|
||||
#
|
||||
# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
|
||||
# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
|
||||
# instead of probing names you assume exist. That mistake cost a wasted PATCH.
|
||||
#
|
||||
# * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
|
||||
# PR would never produce that context.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> updating branch protection on main"
|
||||
jq -n '{
|
||||
enable_push: false,
|
||||
enable_status_check: true,
|
||||
status_check_contexts: ["gate / gate (pull_request)"],
|
||||
required_approvals: 1,
|
||||
block_on_rejected_reviews: true,
|
||||
dismiss_stale_approvals: true,
|
||||
block_on_outdated_branch: false,
|
||||
block_admin_merge_override: true
|
||||
}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
|
||||
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
|
||||
|
||||
echo "==> verifying"
|
||||
api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
| jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
|
||||
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
|
||||
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
|
||||
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
|
||||
and rely on the status check alone.
|
||||
|
||||
Then prove it, rather than trusting the API:
|
||||
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
|
||||
git reset --hard HEAD~1
|
||||
NOTE
|
||||
Loaded 100 of 171 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user