Compare commits

..
Author SHA1 Message Date
jakman85 f341a9b744 Revert "feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex"
This reverts commit 9137bbd0c3.
2026-09-06 20:27:41 -05:00
jakman85andClaude Opus 5 6f61c9622d feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 20:24:51 -05:00
jakman85andClaude Opus 5 3ade1d440f fix(conformance): status.py read its own detection vocabulary as a TODO marker
`has_todo = "TODO" in text` — a bare whole-file substring. The new L0 check
`placeholder-guard-coupling/check.sh` ENUMERATES placeholder idioms as data
(`SEED = ["TBD","TODO","FIXME",...]`) because detecting them is its entire job, so
the harness classified a 32-assertion executable check as a TODO scaffold.

Consequence: R-STATE-03 — SAFETY-CRITICAL — reported `exec=1 scaffold=1` and
`not-derivable`, because status.py requires all of a requirement's tagged tests to
be proven. A safety requirement mis-reported by a substring match, plus a standing
--strict-coverage conflict that a previous engineer worked AROUND by rewording
prose instead of fixing the detector.

Replaced with two REDUNDANT structural signals, derived from the corpus (all 43
tests read), not invented:
  1. `# STATUS: scaffold` with the state as the FIRST token after `STATUS:`. Exactly
     3 tests declare it — onboarding-completeness, alerting, gitops-drift — and they
     are exactly the three real scaffolds. 19 others declare EXECUTABLE/RUNNABLE, so
     a loose `.*scaffold` would have matched prose.
  2. A TODO/FIXME marker CARRYING A MESSAGE in the body, where the body starts after
     the header comment block. A marker that is the ENTIRE content of a quoted
     literal is data, not a note, and is blanked before the scan. Every genuine
     scaffold marker in the corpus carries a message, so mid-body comments still count.

The line-322 intent is preserved exactly: each file is classified on its own
structure, either signal in either file sets scaffold + annotation_conflict, the
`executable: "true"` annotation is consulted only AFTER the body verdict, and
ambiguity still only downgrades. Vocabulary stays TODO/FIXME only — TBD/PLACEHOLDER
are excluded because two checks use them in real executable code.

NON-VACUITY, and the redundancy is the point: deleting the `# STATUS: scaffold` line
from all three real scaffolds still catches all three via the body rule. The decisive
perturbation is the reverse — the exact file that caused the bug, degraded to a
TODO-only body while KEEPING its `STATUS: EXECUTABLE` header, its idiom-naming header
prose and its `executable: "true"` annotation, is caught as a scaffold WITH the
annotation conflict raised. The annotation still cannot outrank the body.

TAGGING: the agent RECOMMENDED KEEPING `R-STATE-03` on placeholder-guard-coupling and
reversed my lean, on a ground I had missed and which is load-bearing:
ci/touched-requirements.sh maps `bootstrap/*` -> R-STATE-03, and `run.sh
--requirements` resolves R-IDs to test dirs THROUGH THESE LABELS. Removing the tag
would reopen the exact merge-gate hole a984442 closed. It also measured the
counterfactual — with the tag removed R-STATE-03 becomes `stale-proof`, not `proven`
— so the tag is not what holds PROVEN at zero.

R-STATE-03 before -> after: `exec=1 scaffold=1` / not-derivable + strict conflict, to
`exec=2 scaffold=0` / not-derivable with an ACCURATE reason (no ledger row for the L0
check; dr-restore is stale-proof).

LEDGER MECHANISM CONFIRMED CORRECT, not worked around. R-STATE-02's node-loss proof
(sha 57d4df8c) went stale because a984442 touched its evidence set — attributed
precisely to the single path `ci/touched-requirements.sh`. Fail-closed and by design.
Re-proving needs a real L3 run recorded from a clean tree at HEAD-or-later: a
live-cluster action.

OBSERVATION FOR LATER (not changed): scan_implementation counts
ci/touched-requirements.sh as implementation evidence for EVERY R-ID it names, so any
edit to that dispatch table invalidates every proof it mentions. Both current
stale/not-derivable verdicts trace to that one file. Fail-closed is the right
direction; whether a CI path-map should be implementation evidence is a separate call.

VERIFIED BY ME: `status.py --selftest` PASS (8 new assertions, both directions);
`--status --strict-coverage` exit 1 -> 0; `--layer l0` 11/11; the three
`STATUS: scaffold` files are exactly the three real scaffolds; kyverno L0 29/29 and
the full L0 stage green IN THE CONTAINER (the agent saw the stage skip on the host,
where the pinned kyverno CLI is absent). Full CI gate PASS. Counts unchanged:
coverage 48/51, scaffold-only 4, PROVEN 0 (stale 1).

Refs R-TEST-01, R-TEST-04, R-TEST-05, R-STATE-03.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 20:24:22 -05:00
jakman85andClaude Opus 5 cb1916f092 feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0
safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS.

R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control
plane" is half authored and half observable.
  l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from
  the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with
  no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init
  founder per environment; odd, quorum-capable server count with tolerance >= 1;
  tls-sans carry the VIP and every server IP; kube-vip leader-elected with
  cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside
  its environment's LB pool and not a node IP.
  l2-behavioral/ha-control-plane: control-plane AND etcd role on every node,
  every node a REGISTERED etcd member, exactly 3 when the cluster says
  CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every
  server's own address — which proves the VIP fronts the API and that every server
  runs an apiserver, not one founder plus two joiners.

  An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's
  LAN and fails against dev (nodes behind the pfSense NAT), making the test a
  property of where it was launched. Hence the in-cluster Job.

R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the
totality/disjointness of the selecting rules, not another per-service check.
  l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the
  same closed set in schema + Kyverno policy + renderer; EXACTLY ONE
  CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn;
  nothing selects `internet` (absence of a rule IS the path); RFC1918 containment
  computed with `ipaddress` plus every address each overlay assigns itself; the
  tunnel default route and blackhole confined to table 51820 behind an ip rule that
  must carry a from/fwmark/iif selector; and the nft chain `policy accept` with
  every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test
  does not assert.

R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable".
  l0-static/auditability-proofs-producible (78 assertions) drives the real
  machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer
  the spec's wording demands (fault injection and DR drill => L3), taken from
  status.py's own verdict; run.sh --requirements must resolve and pass the gate's
  --require assertion; each proof declares a sentinel record_run.py can attribute,
  computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted
  against the real tools — a green JUnit whose log holds only a SKIP token, and a
  ledger row claiming `proven` at the root commit — which must come back not-proven
  and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted,
  not hand-listed.

*** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's
default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel,
so record_run.py could never attribute a pass to them — the isolation and TLS audit
proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared,
and status.py requires all of a requirement's tests to be proven). Each now ends
with an additive terminal sentinel step reached only when every prior step passed.
Verified: all three carry it.

TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this
repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role
defaults, so flipping the template to true still passed — now parses the DaemonSet
template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja
variable while the template carries its NAME, matching nothing and passing
vacuously — templates are now rendered against role defaults first. 30 perturbations
across the three checks, all caught, plus 16 synthetic L2 perturbations via a
fake-kubectl shim.

R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's
widgets.yaml carries only datetime and search, kubernetes.yaml is empty,
automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress
only — no health source is even reachable. It has already drifted (crafty authors a
route with no tile). A test today would go red on both; it must ship WITH the
config fix.

VERIFIED: full CI gate PASS. Neither cluster mutated.

Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 19:35:18 -05:00
jakman85andClaude Opus 5 5ec1460532 docs: DECISIONS-LOG — prod solidified, four cold-start bugs of one shape
Node 2 at full 32GB; storage recovered 17/17 after the cordon that was silently
blocking Longhorn rebuilds; Kyverno/CDI unblocked by satisfying the policy rather
than exempting it; conformance 82.4%->88.2%. Names the recurring class — a thing
that looks applied and isn't — and lists what is staged for one push versus what
waits on the operator.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 18:51:00 -05:00
jakman85andClaude Opus 5 a984442cae harden: pin node timezone to UTC, and TEST the guard/value coupling that bit us
Two debt items, both about making a latent problem impossible rather than fixed.

=== 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) ===

The prod nodes disagreed, measured from k3s's own etcd snapshot annotations:
prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless —
Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes
apart, which is correct staggering — but it makes cross-node log correlation
actively misleading, and this cluster has had two unplanned reboots in two days.

`Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records
"no other service in the repo sets a timezone ... so use the safe default" with
TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago
DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact:
THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than
inventing a convention. UTC also removes the class — no DST discontinuity, node
time == platform time.

New knob `node_baseline_timezone`, overridable per environment via group_vars (the
same seam as node_baseline_backports_kernel) but deliberately NOT restated there:
no environment needs to differ, and a second copy is a second thing to disagree.
Guarded by an assert in the role's existing BLOCKED style before the module runs.

`--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the
offset evidence, and is the whole verification — the play was never run for real:
  k3s-prod-1: -"America/Chicago"  +"Etc/UTC"  changed
  k3s-prod-2: -"America/Chicago"  +"Etc/UTC"  changed
  k3s-prod-3: -"America/New_York" +"Etc/UTC"  changed
The same --check against DEV reports the task `ok` on all three — idempotence
demonstrated against live hosts, in check mode only. `--list-hosts` confirms
`-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take
their OWN var; the wrong flag silently targets dev).

*** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was
deliberately left for the operator to schedule: prod's Longhorn had only just
finished resyncing after a node reboot, and changing a node's clock config during
storage recovery is not an unattended move.

`ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values
(the passphrase is mounted into the dev container) and has leaked the prod
become-password to a transcript once already.

=== 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) ===

On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from
`TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in
bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died,
and `resume` would have enabled velero on prod against a nonexistent hostname:
installs clean, reports healthy, no backups exist. Found by accident. A guard in
one file keyed on the SHAPE of a value in another file, with nothing tying them.

The check exercises the REAL predicate — it does not reimplement it. standup.sh is
not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the
check extracts the predicate's own source lines and sources only those. Faithfulness
is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name),
exactly-one-definition, "extracted block is a verbatim substring", a self-containment
gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent
second copy. If the predicate grows a dependency the check FAILS LOUDLY and names
the remedy rather than degrading to testing a copy — the failure mode it exists for.

Both sides derive from real files: guarded knobs from a taint trace in standup.sh
(`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base
cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent
detector is deliberately NOT the guard: seed idioms UNION the guard's own words with
glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm
cannot make a value stop being placeholder-shaped.

Real tree: 32 assertions, 18 values through the extracted predicate, PASS.

NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the
read-only spec repo untouched). My first perturbation tripped the self-containment
rail instead of the anchoring property, so I redid it shape-preserving — substring
globs -> prefix globs, nothing else — and it failed on exactly the right property,
naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05
failure exactly". Control: unperturbed copy exits 0.

=== 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) ===

The path->requirement map had no `bootstrap/*` entry, so a change touching
standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's
requirements and silently excluded the new check — the one that exists to catch
exactly the 2026-09-05 class. A safety net with a hole where the incident was.
Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths.

VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check.
Coverage unchanged at 45/51 (the new check tags existing requirements).

Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 08:47:08 -05:00
jakman85andClaude Opus 5 da3e9cc3f3 fix(kyverno): unblock CDI imports by SATISFYING the label requirement, not exempting it
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:

  validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
    require-auth-mode-label     (R-IDP-02)
    require-egress-class-label  (R-NET-05)

Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.

MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.

Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.

THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.

REJECTED, and the first one is a trap worth recording:
  * PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
    all controllers run `--enablePolicyException=false` (verified live on prod:
    admission, background and reports controllers all carry the flag). An exception
    would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
    cluster-wide capability change for no gain over the mutate.
  * `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
    match forces `background: false`, which would drop background scanning and
    policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
    class. It also leaves the importer with no labels at all — the unlabelled
    fall-through that require-egress-class-label's own header calls a fail-open leak.
  * name-prefix / CDI-label matching — attacker-choosable, and proven walkable.

TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
  userinfo cdi-sa   -> mutate pass, require-egress-class pass, require-auth-mode pass
  userinfo ordinary -> mutate SKIP,  require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.

platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.

RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.

FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.

CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.

VERIFIED: full CI gate PASS. Nothing applied to any cluster.

Refs R-IDP-02, R-NET-05, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 08:41:05 -05:00
jakman85andClaude Opus 5 04569abd78 docs(prod): node-maintenance runbook — a failed drain leaves a cordon that stops self-healing
Two unplanned node events (prod-3 ungraceful power-down; prod-2 memory upgrade)
produced one finding worth more than either incident.

A FAILED `kubectl drain` STILL CORDONS, and Longhorn's
`disable-scheduling-on-cordoned-node` is `true` — so a cordoned node gets no
replica scheduling. Observed after prod-2's reboot: ALL 17 VOLUMES sat
attached/degraded with prod-2 holding 2 of 18 replicas and ZERO rebuilds in
flight. Nothing was driving recovery, every pod was Running, and no alert fired.
`kubectl uncordon` started rebuilds immediately; converged 17/17 healthy in ~6
minutes (measured: 9→10→11→13→14→16→17 over 5 polls).

So uncordon is not tidying up — it is what re-enables storage redundancy. A green
pod list proves nothing about it.

DRAIN CANNOT COMPLETE ON THIS CLUSTER, and the two usual flags do not fix it.
`--ignore-daemonsets --delete-emptydir-data` clears the reported objections and
then hangs: all three instance-manager PDBs report ALLOWED DISRUPTIONS = 0 and
node-drain-policy is block-if-contains-last-replica. Requesting replica eviction
does not rescue it either — with 3 nodes and 3 replicas there is no fourth node to
move a replica to, so eviction can never complete. That is structural. Documented
options in preference order: just reboot (proven twice here; the 600s
replenishment interval means a quick reboot triggers no full rebuild); or scale
down the workloads holding engines on that node so the PDBs free up. Changing
node-drain-policy to always-allow is advised against at 3/3.

Also recorded: check the kube-vip leader before rebooting a server — it holds the
API VIP via a lease and is a DaemonSet, so a DRAIN does not move the VIP but a
REBOOT does, and kubectl blips for seconds. Plus a post-maintenance checklist
whose first item is verifying no cordon remains, and whose memory check exists
because this hardware silently strands RAM on the wrong kernel (§0b).

KNOWN, NOT FIXED: pod distribution does not rebalance — 52/10/50 after prod-2's
cordon+reboot. For the many single-replica apps here a spread constraint would
change nothing (one replica, one node). It self-corrects as pods restart. Recorded
because it concentrates failure: all three CNPG databases currently run on prod-1,
so losing that node takes Keycloak, Nextcloud and Semaphore together. The fix is
database HA (instances: 2), a resource decision the operator owns.

Header refreshed: phases A/B/C are done, 12/13 tiers, 112 pods, all three nodes at
full 32 GB after node 2's DIMM upgrade.

Refs R-STATE-01, R-STATE-02, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 08:35:41 -05:00
jakman85andClaude Opus 5 21e111b08c feat(devcontainer): dev.sh can target prod — and cannot silently target the wrong cluster
Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.

WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.

Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).

    infra/devcontainer/dev.sh run kubectl get nodes                # dev
    CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # prod

Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.

New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.

FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).

VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
  default (no selector)  -> home.keithjackson.dev        (DEV, unchanged)
  CLUSTER=prod           -> prod.home.keithjackson.dev   (PROD)
  contexts               -> k3s-dev, k3s-prod; NO `default` context survives
  exit codes             -> invalid CLUSTER=2, valid dev/prod=0
  standup.sh             -> `--kubeconfig` still wins over the new default
                            (preflight against prod: target confirmed 192.168.2.9)
  credentials            -> dev.kubeconfig and prod.kubeconfig byte-intact and each
                            still reaching 3 nodes after the agent's move-aside test
  git                    -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.

NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.

DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.

Closes the PHASE1-STATUS "Known gap".

Refs R-ENV-02, R-TEST-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 16:38:32 -05:00
jakman85andClaude Opus 5 577c62136b docs: DECISIONS-LOG — PM autonomous session handoff
Backups moved to R2 (blocked on the operator's token), CDI cold-start deadlock
repaired in the graph, six L2 tests made environment-agnostic, conformance
82.4% -> 88.2%. Records the guard I broke and how it was caught, the R-TLS-02
'revoked' gap with the prune/rate-limit trade-off that makes it an operator call,
and the merge preconditions for both the CDI split and the R2 switch.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 14:17:41 -05:00
jakman85andClaude Opus 5 fe45d0f5c5 feat(conformance): cover R-TLS-02, R-ENV-01, R-ENV-03 — 42/51 -> 45/51, and find a real gap
DECISIONS-LOG records Phase 7 being opened knowingly while the uncovered set
included R-ENV-01/02/03 — the very requirements prod provisioning serves. Two of
those three are now covered, plus R-TLS-02.

  tls-lifecycle-automated          R-TLS-02  L0 (in the merge gate)
  cert-lifecycle-automated         R-TLS-02  L2 (authored, NOT executed)
  dev-nodes-provisioned-as-code    R-ENV-01  L0
  prod-baremetal-not-dev-surface   R-ENV-03  L0

Coverage 42/51 (82.4%) -> 45/51 (88.2%), 6 gaps, 0 safety-critical. Remaining:
R-C2-01, R-C2-02, R-NET-04, R-PLAT-01, R-SEC-03, R-STATE-04. Verified through the
real runner: --layer l0 7/7 PASS, the three new ones 3/3 PASS, --strict-coverage
exit 0 with 0 label/annotation mismatches.

*** R-TLS-02 "REVOKED" IS NOT SATISFIED — A REAL GAP, NOT A TEST DEFECT. ***
The spec requires certificates be "issued, renewed, AND REVOKED without manual
steps", and docs/08 reads revocation as "removing/rotating a certificate is a
declarative change". That needs enableCertificateOwnerRef: true;
platform/cert-manager/helmrelease.yaml does not set it (chart default false).
Verified read-only on BOTH live clusters: gateway/wildcard-home-tls has ZERO
ownerReferences, so deleting the Certificate would leave the private key serving
indefinitely. The L2 test asserts the requirement AS WRITTEN and will fail until
this is decided. It was deliberately NOT softened — weakening a test to match the
implementation is how a suite stops meaning anything.

NOT FLIPPED UNATTENDED, and the reason is specific rather than caution: the
wildcard Certificate lives under ./platform, which runs prune: true. With owner
refs ON, any transient prune of that Certificate — exactly what happened to the
CDI StorageProfiles hours ago — would delete the SERVING TLS Secret for every
host, and reissuance is bounded by Let's Encrypt's duplicate-certificate limit
(5/week per identical name set), which could leave a cluster with no trusted cert
for a week. Security gap vs a plausible week-long TLS outage is an operator call.

NON-VACUITY DEMONSTRATED, which is the part that makes these worth having:
7/7 perturbations fail for R-TLS-02, 8/8 for R-ENV-01, 13/13 for R-ENV-03, and
each restored copy passes. Perturbations were applied to a throwaway copy; neither
this repo nor the read-only spec repo was modified.

TWO CHECKS INITIALLY PASSED WHEN PERTURBED and were rewritten — the exact
degrade-to-no-op failure this repo keeps hitting. The first draft asked "is this
file's DIRECTORY referenced by a kustomization", which a directory reference always
satisfies, and keyed Secrets by name alone, so external-dns's copy of
`cloudflare-api-token` satisfied cert-manager's reference. Now a recursive
kustomize resource-graph walk from each Flux path, and Secrets keyed by
(namespace, name). Both bite.

SECURITY NOTE, recorded because this project has already leaked a password once:
`ansible-inventory --list` PRINTS DECRYPTED VAULT VALUES — it emitted the prod
become-password to stdout during exploration, since the vault passphrase file is
mounted into the dev container. Both new checks use only --syntax-check and
--list-hosts, and say so in a comment. Confirmed no plaintext value reached disk:
vault.yml is still AES256-encrypted and the tree has no assigned value.

WHAT IS NOT PROVEN, per requirement: R-TLS-02 "revoked" (above), and issuance +
observed renewal await the unexecuted L2. R-ENV-01 cannot prove the running VMs
came from this module (needs OpenTofu state) — it proves the four lifecycle stages
exist as code and cover every dev node, and that the chain RESOLVES under real
ansible-playbook. R-ENV-03 cannot prove the HARDWARE (the K8s API exposes no
"am I virtual" fact; mini-PC/bare-metal is operator-attested) — it proves prod is
AUTHORED as 3 directly-reached nodes with no IaC creating them, sharing dev's
playbooks and roles, with no prod-only manifest, branch or role anywhere.

VERIFIED BY ME: coverage delta and the 3/3 L0 pass through the real runner; the
zero-ownerReferences finding on both clusters; that the new checks avoid
ansible-inventory; that no credential material reached disk. Full CI gate PASS.

Refs R-TLS-02, R-ENV-01, R-ENV-03, R-TEST-01..05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 14:17:03 -05:00
jakman85andClaude Opus 5 8f9b2e4270 feat(backup): s3check proves a real round trip — and revives a guard I had killed
*** THE BUG, AND IT WAS MINE. *** The placeholder guards were `case "$s3" in
""|TBD-*)` — a PREFIX glob. Two commits earlier (32d6766) I changed the
placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`, which does NOT start with
`TBD-`. Verified: fnmatch('https://TBD-R2-...', 'TBD-*') is False. All three
guards silently stopped matching, so **step_resume would have resumed velero on
prod straight onto a nonexistent hostname** — velero installs, reports healthy,
and no backup exists. That is precisely the failure the guard was written to
prevent, disarmed by a value change in a DIFFERENT file with nothing tying the
two together. Replaced with substring-matching s3_is_placeholder in preflight,
s3check and resume.

s3check rewritten from a reachability ping into a real proof. Per credential, from
a throwaway in-cluster pod (pod egress is the only egress that matters — the
workstation sits on a different network side and a host-side reading has already
misled this project once):
  1. the credential LOADS AND AUTHENTICATES in the exact shape its consumer reads
     — Velero's INI mounted from the `cloud` key, CNPG's discrete env keys;
  2. the bucket EXISTS and is LISTABLE under this environment's own prefix, via
     ListObjectsV2 — the op Velero's BSL sync and Barman's WAL scan actually use.
     ListBuckets deliberately avoided: a bucket-scoped R2 token may not grant it;
  3. PUT -> GET -> sha256+length compare -> DELETE -> confirm-gone. A HEAD would
     pass with a read-only token and with every write being rejected;
  4. the R2 checksum trap probed ON PURPOSE by repeating the PUT with checksums
     enabled, and named either way.
Buckets and prefixes are DERIVED (velero's from the authored HelmRelease via yq,
cnpg's parsed from CNPG_BACKUP_PATH), not restated.

Fails closed, with no "unknown, carrying on" arm left: no pod, unfinished pod,
empty log, missing/duplicate/malformed verdict, any failed assertion, a claimed
PASS with failures>0, an assertion count below the caller-computed minimum, an
OK-line count disagreeing with the claimed count, or a PASS from a pod that ended
Failed. Failures are classified by cause — no egress, NoSuchBucket,
InvalidAccessKeyId, AccessDenied (token not scoped to BOTH buckets),
InvalidArgument -> the checksum trap.

`amazon/aws-cli` pinned rather than `minio/mc`: mc lacks sed/awk (which already
broke one probe) AND cannot send checksum headers, so it cannot test for the trap.

TIER_ORDER now includes platform-config-cdi, between platform-config and the leaf
tiers. Without it `resume` returned success without ever waiting on the tier that
guarantees the CDI StorageProfiles — and apps-vms depends on them: unifi
provisions on longhorn-vm, where an unapplied profile means CDI infers RWO instead
of the authored RWX, permanently non-live-migratable.

CREDENTIALS: one key, four committed Secrets, two shapes — and NO manifest change
is needed, only encrypted values. platform/backup/set-r2-credentials.sh writes all
four with `sops set`, which encrypts IN PLACE, so no decrypted Secret ever exists
in the working tree to be committed by accident — strictly better than staging a
plaintext template. Prompts with `read -rs`, rejects anything outside the S3
credential charset (a paste that swallowed a newline authenticates nowhere), never
prints a value, and verifies by decrypting back and comparing fingerprints.
`--check` is read-only. The MinIO key is RETAINED — it is what reads the
pre-migration backups.

THE ACCOUNT ID IS ALREADY IN THE REPO: infra/tofu/backend.r2.hcl commits
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com for the tofu
state bucket. If the backup buckets go in the same account, only the two token
values are needed. The placeholder is KEPT anyway — fail-closed beats convenience,
and the guard above just demonstrated why.
Corroborating the checksum fix independently: infra/tofu/backend.tf already carries
`region = "auto"` and `skip_s3_checksum = true # R2 rejects AWS trailing
(streaming) checksums`. This repo had learned both R2 facts for tofu state already.

VERSION FINDINGS, documented not measured against R2: plugin v1.14.2 documents
checksumAlgorithm defaulting to CRC32, so the authored "" is load-bearing;
Velero's kopia backend has NO checksum knob, so BSL metadata and PVC-data
compatibility are SEPARATE questions; and in the live CNPG 16.4 image,
boto3/botocore 1.35.60 is below the 1.36 line that turned on default request
checksums — so CNPG should be clear today, but barman-cloud exposes no escape
hatch, making a CNPG image bump an R2 re-verification event.

VERIFIED BY ME, independently of the agent: the dead-guard glob (fnmatch False);
the placeholder now refused on prod before any pod is created; a full round trip
against dev's MinIO — 7/7 assertions on BOTH credential paths, byte-identical,
pod Succeeded, nothing left behind, checksum trap correctly reported absent; all
four Secrets and the committed account id present. Full CI gate PASS.

STILL UNPROVEN AND SAID SO IN THE DOC: Velero-to-R2 (metadata or kopia data),
CNPG-barman-to-R2, and any restore from R2. R-STATE-03 is PROVEN MinIO-ONLY and
its ledger row predates this move. infra/BACKUP-R2.md §5 is the re-proof sequence
and it ends in a RESTORE, not an upload.

Refs R-STATE-03, R-IDP-05, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 13:24:26 -05:00
jakman85andClaude Opus 5 59ffbaf11a feat(conformance): L2 tests derive their environment from the cluster, not a literal
Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS,
not reconciled manifests, so postBuild substitution never touched them — correct
while dev held the apex, wrong the moment prod came up on
prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod".

Each test now asks the cluster what it is, via a test-level chainsaw binding over
flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so
a test cannot disagree with the cluster it runs on:

  bindings:
    - name: baseDomain
      value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN)

Chosen over a flag or env var because there is nothing to forget at run time, and
test-level scope covers every step (step outputs/bindings are step-scoped).

THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also
hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so
both were wrong for prod regardless of hostname. Replaced with assertions on the
OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the
public record must equal the LB IP the Service was actually assigned — which is
strictly STRONGER than the old range check as well as environment-independent.
sso-forward-auth held the literal only in prose (it talks over Service DNS).

PROVEN, NOT ASSERTED — and the positive control is the important half:
  * expressions extracted verbatim from the tracked files resolve to the correct,
    DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53,
    prod prod.home.keithjackson.dev / 192.168.2.20);
  * the same assert with the HARDCODED apex passes on dev and FAILS on prod
    ("Invalid value: *.prod.home... Expected: *.home..."), so the harness is
    provably not blind — that is exactly the drift removed;
  * chainsaw lint VALID on all six; run.sh --lint clean over the whole tree.

END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding
produced conformance-dns.home.keithjackson.dev, external-dns published it, and the
probe Job asserted the record against the assigned LB IP. This is the first time
one of these tests has been executed since parameterization, and it exercises the
full path rather than just the binding.

FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It
reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record
is reaped later by external-dns's own --interval=1m sync. A zone query immediately
after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default
TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove
cleanup happened, and if external-dns were suspended the record would persist
silently. Verified separately that it clears.

conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now
LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five
tests. Commented so it is not "tidied" later.

l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven
because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the
text now says both — that the R2 move (32d6766) is what closes it, that this test's
pass is against MinIO, and that the ledger row proving R-STATE-03 PREDATES the move,
so off-host DR cannot be claimed until the drill is re-run against R2.

NOT VERIFIED, plainly: no full L2 suite run against prod, so "green against prod"
remains unproven — what is proven is that the tests now derive the CORRECT
expectation for whichever cluster they are pointed at. Whether prod SATISFIES them
(split-horizon, LE wildcard, Grafana's issuer) is untested.

VERIFIED: full CI gate PASS. flux-postbuild-substitution 260 assertions,
render-golden 10/10.

Refs R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-ENV-02, R-DNS-01/02/03.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 12:05:05 -05:00
jakman85andClaude Opus 5 765ad032db fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.

The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.

THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.

Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.

apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.

THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
  longhorn-vm (we override)     spec RWX/Block  -> status ReadWriteMany/Block
  longhorn-static (no override) spec unset      -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.

MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).

substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.

VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.

KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 11:53:45 -05:00
jakman85andClaude Opus 5 32d67660a6 feat(backup): move off-cluster DR to Cloudflare R2 (operator decision 2026-09-05)
"That's where backups will go for the time being until a better on-prem backup
solution is created." Two reasons beyond the (negligible) cost:

  1. Prod CANNOT reach the on-Proxmox MinIO. Prod is on the SOHO network
     192.168.2.0/24 — pfSense's WAN — while that MinIO sits inside pfSense's LAN.
     Measured from a pod inside prod: connection timed out after 15s. R2 needs no
     firewall change.
  2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's
     "off-cluster" backup shared a physical failure domain with dev — one host
     loss took the cluster AND its backups. R2 is genuinely off-site.

COST, measured rather than estimated (read straight out of the buckets):
  velero  4.0 GiB / 6,264 objects  (7d retention; 8.65 GB live -> kopia dedup)
  cnpg    211 MiB / 1,755 objects  (keycloak 66Mi/331, nextcloud 87Mi/1329,
                                    semaphore 58Mi/95; ~7d of a 30d window)
Both clusters project to ~13-15 GiB and ~30k writes/month => pennies on R2, and
free egress means restore REHEARSALS cost nothing, so R-STATE-03 has no financial
disincentive to stay proven. Wasabi/Hetzner are 50-200x worse here purely because
of 1 TB minimums; Glacier tiers are a trap (90-day minimum billing against a
7-day Velero retention plus early-delete fees on 30-day WAL pruning).

*** THE R2 GOTCHA, WIRED UP FRONT: checksumAlgorithm: "" ***
velero-plugin-for-aws v1.14.2 bundles an AWS SDK that sends
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT. R2 REJECTS
those with HTTP 400 InvalidArgument, so every backup and every kopia blob upload
would fail. Set unconditionally — MinIO ignores it, so it needs no fork.

`region` was hardcoded us-east-1 inside the Velero HelmRelease: correct-by-
accident for MinIO (which ignores it) and WRONG for R2 (which demands exactly
`auto`). Now the S3_BACKUP_REGION knob.

PREFIXES ARE NOW SYMMETRIC (dev/prod), reversing the reasoning from two commits
ago. While the target was MinIO, dev HAD to keep the bucket root or it would
orphan the backups its proven restore depends on. R2 starts BOTH clusters on an
empty bucket, so there is no history to orphan — and leaving the bucket root
unused means a misrendered prefix lands somewhere obviously wrong instead of
silently colliding with a real environment.

NEITHER OVERLAY PINS THE ENDPOINT ANY MORE — a deliberate reversal of this
repo's "prod must never inherit base" rule. That rule exists because base
defaults are dev's LAN addresses; R2 is a single environment-INDEPENDENT endpoint,
so under R-ENV-02 it belongs in base, and two copies of one URL is a rotation
waiting to drift. Isolation comes from the path knobs, not the URL. Caught while
verifying: dev's overlay DID still pin MinIO, so dev would have taken the new
`dev` prefix while writing to the old endpoint — orphaning its MinIO backups for
no benefit. That is why the endpoint and the prefixes must move in ONE commit.

*** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts empty:
Velero writes a fresh kopia repo, CNPG takes new base backups, the 30-day Barman
window rebuilds from zero. The MinIO copy is NOT deleted and stays restorable by
pointing the endpoint back. Do not merge while dev's proven restore is the only
recovery path you rely on.

STILL BLOCKED ON THE OPERATOR: the endpoint is the loud placeholder
https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com. Creating the R2 buckets and
an API token needs Cloudflare account access I do not have (the token in this
repo is Zone:DNS scoped). Velero+R2 and CNPG-barman+R2 are also UNPROVEN here and
must be verified by an actual backup AND restore, not just an upload.

VERIFIED: full CI gate PASS. Rendered both overlays — dev dev/s3://cnpg/dev,
prod prod/s3://cnpg/prod, region auto, one shared endpoint.

Refs R-STATE-03, R-ENV-02, R-IDP-05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 11:29:22 -05:00
jakman85andClaude Opus 5 9063e53ce2 docs(prod): prod verified end-to-end through the real resolver — dnsmasq profile installed
Operator installed infra/workstation/dnsmasq/install.sh. Verified afterwards:

RESOLUTION (live NetworkManager dnsmasq, `getent hosts`):
  prod.home / gitea.prod / keycloak.prod / grafana.prod -> 192.168.2.20
  unifi.prod -> 192.168.2.21     crafty-mc.prod -> 192.168.2.22
  gitea.home / home.keithjackson.dev -> 192.168.2.50   DEV, UNCHANGED

HTTPS with NO --resolve override and NO -k, all landing on 192.168.2.20:
  apex 200 · gitea 200 · headlamp 200 · semaphore 200
  keycloak 302 (login) · grafana 302 (SSO) · nextcloud 302 · pihole 302

CERTIFICATE, confirmed by SNI — this is the check that actually settles the
earlier confusion, because the resolver bug made dev's cert look like a broken
prod cert:
  subject CN=*.prod.home.keithjackson.dev
  issuer  C=US, O=Let's Encrypt, CN=YR1   (LE production)
  SAN     *.prod.home.keithjackson.dev, prod.home.keithjackson.dev
  valid   2026-09-05 .. 2026-12-04
Dev checked in the same pass still serves CN=*.home.keithjackson.dev.

Also noted so it is not misread as a fault: `curl --cert-status` reports
"(91) No OCSP response received" because Let's Encrypt retired OCSP responders
in 2025 — there is nothing to staple. The plain requests validated normally.

Prod is now reachable and trusted from the workstation by name. Outstanding items
are unchanged: MinIO routing (prod is on pfSense's WAN), the apex flip being the
same event as the legacy-prod 443 cutover, and delivering the backup-tenancy
commits to prod via a c8c1d29-based branch.

Refs R-DNS-02, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 11:11:53 -05:00
jakman85andClaude Opus 5 6d62aca0d2 feat(workstation): prod dnsmasq profile — and CORRECT the topology I had wrong
OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows
prod "a pfSense override" throughout this repo. Both halves of that were wrong:

  - It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the
    one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/
    dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is
    dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50`
    matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new
    prod.home.* tree and answered it with dev's address.
  - The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router
    network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN
    192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the
    rules below use prod's real VIPs and need no NAT hop, unlike dev's .50.

New infra/workstation/dnsmasq/ — authored, not click-ops:
  20-prod.home.keithjackson.dev.conf   prod.home.* -> 192.168.2.20 (Gateway),
                                       unifi -> .21, crafty-mc -> .22 (L4 VIPs)
  install.sh                           installs + reloads; refuses non-root

dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT
editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway
dnsmasq on 127.0.0.1:5354 loaded with both profiles —
  prod.home / gitea.prod / keycloak.prod -> 192.168.2.20
  unifi.prod -> 192.168.2.21   crafty-mc.prod -> 192.168.2.22
  gitea.home / home.keithjackson.dev     -> 192.168.2.50  (DEV, UNCHANGED)
`dnsmasq --test` on the new file: syntax OK.

install.sh pins the destination because getting it wrong is SILENT: a file in
/etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses
`systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read
its config on SIGHUP, so a HUP would look like it worked and not apply anything.

WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a
*.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev
(one label) so the prod apex returns 200 over VALID TLS, while
keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name
error. It reads as a broken prod cert; it is the wrong server answering. I made
exactly that misreading during verification.

Also recorded, because it changes what the apex flip IS: the SOHO router NATs
inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S
production environment, which must stay live until the switch is deliberately
made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the
legacy-prod cutover, and P as written does NOT cover repointing that NAT. The
two must be planned together or the apex resolves to k3s while inbound 443 still
lands on the legacy box. Phase C touches none of it: prod's Gateway is
192.168.2.20:443 (a different address) and its cert is DNS-01, needing no
inbound 443. And the MinIO timeout is now correctly described as expected
topology — crossing INTO pfSense from its WAN — not a misconfiguration.

VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1).

Refs R-DNS-02, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 11:08:54 -05:00
jakman85andClaude Opus 5 b80e8eef82 feat(backup): prod shares dev's MinIO — plus the path tenancy that makes it survivable
Operator ruling: "the S3 backup location is the same as Dev." Prod's
S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder
is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would
inherit the same value silently, and a shared backup target must be visible in
the overlay that owns it.

SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS
was per-environment, and enabling prod against dev's MinIO would have been the
2026-09-04 external-dns incident again, in a worse place:

  VELERO — bucket `velero`, NO prefix, and both clusters run schedules named
    `daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL
    into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would
    have adopted dev's backups and then deleted them on schedule.
  CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending
    serverName, which defaults to the CLUSTER NAME. Both environments have
    keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive
    WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one
    Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is
    PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not.

Two knobs, authored once, environments differing only by scoped overrides
(R-ENV-02):
  VELERO_BACKUP_PREFIX  dev ""          | prod "prod"
  CNPG_BACKUP_PATH      dev "s3://cnpg" | prod "s3://cnpg/prod"

THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live
at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very
backups its proven restore depends on — the same reasoning that keeps dev's
EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve.

*** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. ***
Measured from a throwaway pod INSIDE the prod cluster (the workstation proves
nothing about pod egress, and a workstation-side reading has already misled this
project once): curl http://192.168.1.9:9000/minio/health/live → connection timed
out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO.
OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN
regardless — the token IS defined so strict substitution succeeds — while every
write fails at RUNTIME. A green apps-wave-0 does not mean backups work.

New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster
pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before
authoring it into cluster-vars. That is how the unreachability above was
established rather than assumed.

VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions,
and neither new knob appears in the dead-knob advisory list, so both are really
consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's
prefix carry the tokens; dev resolves to its existing layout, prod to the nested
paths.

DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also
carries the DNS inversion, which must NOT reach prod. Delivering needs a
c8c1d29-based branch with only these commits — no urgency while the routing is
still closed.

Refs R-STATE-03, R-ENV-02, R-IDP-05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 10:59:50 -05:00
jakman85andClaude Opus 5 e6bcecc78c docs: DECISIONS-LOG — overnight prod standup handoff
Prod is up: 12/13 tiers Ready, LE wildcard issued, 113 pods, dev's 57 DNS
records byte-identical throughout. Records both cold-start deadlocks (Kyverno
CRDs, CDI StorageProfile) as one root cause, the external-dns footgun and the
prove-before-acting pattern that disarmed it, three bugs in my own verification,
the pfSense wildcard now shadowing prod, and what remains for the operator.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 10:06:42 -05:00
jakman85andClaude Opus 5 acb5cabdad fix(standup): preflight must always report — it was an env fork in disguise
preflight `die`d when EXTERNALDNS_TXT_OWNER_ID was empty, the sentinel, or
`default`. Two problems, found by pointing it at dev:

1. `default` is DEV'S LEGITIMATE owner id — its live records are registered
   under it and cannot be renamed without orphaning them. Dying on it meant a
   script whose entire claim is "environment-agnostic, no dev/prod fork"
   (R-ENV-02) could never run against dev at all. That is a fork in disguise.

2. Dev's value is currently EMPTY, because dev runs dev-live@a4a3c2e — based on
   main@4809199, which predates the commit that added the knob. That is dev's
   long-standing status quo, not a fault. The die hid the ENTIRE remaining
   health report behind it, which is backwards for a diagnostic.

preflight is now a pure diagnostic: it always finishes and reports, and the only
thing it refuses over is the --expect-server target guard, because that one
means you are pointed at the wrong cluster. Bad owner-id values are reported
with a warning explaining why each is dangerous and for whom.

THE REFUSAL MOVES, IT DOES NOT DISAPPEAR: step_externaldns — the MUTATING path,
where acting on a bad value is what actually deletes another cluster's records —
still dies on empty, sentinel and `default`. Verified both halves after the
change: prod preflight still reports "distinct and non-default: OK", and the
mutating guard is intact.

Confirmed with it: dev is 13/13 Kustomizations Ready on dev-live@a4a3c2e, all
HelmReleases Ready, nothing suspended — healthy throughout prod's standup.

Refs R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 10:06:01 -05:00
jakman85andClaude Opus 5 4f37fc375e feat(prod): STANDUP COMPLETE — 12/13 tiers Ready, LE wildcard issued, dev untouched
Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home
Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway
VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under
prod.home.keithjackson.dev, every TXT registry entry owner=prod.

End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway:
apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO).

DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup
baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD
placeholder, and resuming it would create a BackupStorageLocation that can never
reach a bucket: a tier reporting healthy while backups silently do not exist.

TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check
worthless. Recording them because a check you cannot trust is worse than none —
it trains you to skip the one alarm that matters.

1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the
   DNS baseline was discarded between steps and `verify`'s before/after
   comparison — the check that detects damage to the other cluster's records —
   could never run across invocations. Now repo-relative (bind-mounted, so it
   persists on the host) and gitignored as a run artifact.

2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED",
   listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation —
   the host ordered `*.keithjackson.dev` differently from the container, so a
   diff of two identical sets reported changes. Proved it was noise by
   re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips
   CRs and sorts under LC_ALL=C so ordering is reproducible everywhere.

ACTION REQUIRED (operator, pfSense): the local override is a wildcard on
*.home.keithjackson.dev and now also swallows the new prod.home.* subtree,
resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly
returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT
from behind pfSense. Narrow the override, or add one pointing prod.home.* at
192.168.2.20.

That override is also the trap that produced a false healthy reading during the
2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's
*.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label)
so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels)
fails with a certificate-name error. I hit this myself during verification and
mistook it for a broken prod cert. Never verify DNS or TLS here through a local
resolver.

Refs R-ENV-02, R-GITOPS-01, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 10:02:20 -05:00
jakman85andClaude Opus 5 1b056dc9d9 fix(prod): break the CDI cold-start deadlock, and make preflight catch its class
`platform-config` stopped dead on prod:

    StorageProfile/longhorn dry-run failed: no matches for kind
    "StorageProfile" in version "cdi.kubevirt.io/v1beta1"

A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR *and*
platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's
CRD does not exist until cdi-operator has reconciled that CR — and Flux
server-side dry-runs EVERY object before applying ANY of them, so the unknown
kind aborts the whole tier, the CR is never applied, the CRD is never created,
and platform-config can never converge. Every tier below it stays blocked.

Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order
a resource against a CRD that one of its own siblings is responsible for
creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been
there for months — and fatal on a cold one. Observed on prod: cdi-operator
Available for 12h, no CDI CR, no storageprofiles CRD.

`crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving
every other object to Flux. The CR carries no ${} tokens, so applying it here
cannot diverge from what Flux would render. It refuses if the `cdi` namespace is
absent (platform has not run yet) rather than applying into nothing, waits for
the CRD, and requires it to reach Established. On prod the CRD appeared in 10s
and platform-config was unblocked.

`preflight` now reports the CRDs that later tiers apply resources against
(storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi
namespace when one is missing, so this is caught in seconds instead of after
`resume` burns its 900s wait discovering it. It also names the stale-discovery-
cache variant, which presents identically but clears on a forced reconcile.

THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move
cdi-storageprofiles.yaml out of platform-config into its own Kustomization with
dependsOn: platform-config, so the CR lands one reconcile before the resources
needing its CRDs. That edits the reconciliation graph both clusters share, and
prod currently reads a pinned revision from the bootstrap Gitea while dev is
mid-migration on a separate branch — so it wants doing deliberately, not at
02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any
cold cluster, and PROD-STANDUP.md says so.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 08:05:46 -05:00
jakman85andClaude Opus 5 4160fee6ce feat(prod): standup.sh + PROD-STANDUP.md — the complete standup sequence, in git
The last steps of a prod bring-up were living in shell history: resume the
tiers in the right order, clear a wedged HelmRelease, fix external-dns without
deleting the other cluster's DNS. That is the same failure the Kyverno CRD
pre-apply had, so it gets the same treatment.

bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six
idempotent steps: preflight, crds, unwedge, externaldns, resume, verify.

THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against
the live API server before any step runs; there is no override flag. Dev is
live and shares the keithjackson.dev Cloudflare zone with prod, and on
2026-09-04 prod deleted every one of dev's public records. Negative control
run: prod's kubeconfig with dev's server URL refuses and changes nothing.

preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base
sentinel, or `default` (dev's owner) — the exact condition that caused the
incident. It also surfaces SUSPENDED HelmReleases separately, because a
suspended release keeps its last Ready condition and a stale `Ready=True`
otherwise hides that Flux is not managing it at all (external-dns today).

externaldns is the dangerous step, and the ordering IS the mitigation. A Helm
upgrade resets replicas to 1 and applies new args in ONE operation, so there is
no window to check the result first; if --txt-owner-id were missing the pod
starts as owner `default` and syncs away dev's records. So the step PROVES THE
OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and
assert --txt-owner-id=<owner> appears, aborting if not. Rendering is
deterministic and touches nothing, which turns a race into a precondition. Then
snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if
wrong), and diff the zone.

resume walks the dependency order and STOPS at the first tier that does not go
Ready — a partial understood bring-up beats a forced green. velero is skipped
while S3_BACKUP_ENDPOINT is the TBD placeholder.

TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming:

1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22
   — the policy-report types are in the wgpolicyk8s.io group. A preflight that
   under-reports reads as "the pre-apply failed" when it succeeded. Fixed and
   re-run: 22.

2. The documented node-health invocation used -e k3s_hosts=k3s_prod and
   SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep
   their own target var (node_health_hosts, memattr_survey_hosts,
   vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped
   elsewhere — a good design I did NOT change. The hazard is that the wrong flag
   does not error, it falls through to the dev default. Caught only by
   --list-hosts; the doc now leads with that warning and a dry-run command.
   Verified after fixing: node-health and all six site.yml plays target prod.

.claude/settings.local.json — permission rules for the above. Every entry
hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible
groups), so the permission strings are themselves prod-scoped and cannot be
turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which
would have been blanket cluster access wearing a narrow-looking mask.

Refs R-ENV-02, R-GITOPS-01, R-SEC-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 00:37:48 -05:00
jakman85andClaude Opus 5 7432012713 docs(prod): platform tier up 8/9 — and TWO blockers, one of them an armed footgun
Unsuspended prod's `platform` after pre-applying the Kyverno CRDs.
8/9 HelmReleases Ready, 76/76 pods Running (cert-manager, CNPG, Cilium,
Longhorn, kube-prometheus-stack, Loki, Alloy, external-dns). `platform` itself
is NOT Ready and the 10 downstream tiers stay blocked.

THE CRD PRE-APPLY WORKED and is now proven against a live API server: 22 CRDs
Established, a second run a clean no-op (rc=0), reproduced under the PINNED
toolchain (helm 3.16.0) matching the 3.20.0 run byte-for-byte. It also confirmed
the predicted failure mode: Kyverno's Deployments had 55 RESTARTS EACH over 4.5h
and stabilised within minutes of the apply — controllers with no API types.

BLOCKER 1 — Kyverno's Helm release is wedged in `uninstalling`. The install
timed out *because* of that crash loop; install.remediation.retries:3 then fired
an uninstall which ALSO timed out, leaving sh.helm.release.v1.kyverno.v1 at
status=uninstalling. helm-controller cannot advance from there, so `platform`
(wait: true) never goes Ready. The Deployments themselves are healthy at the
right chart version — this is stale Helm bookkeeping, not a broken workload.
Fix needs `helm uninstall` + `flux reconcile --force`; destructive, so NOT run
unattended. Safe here because prod carries no workloads and a cold install
proves the bootstrap path. The pre-applied CRDs live OUTSIDE the release and
survive the uninstall — precisely why the pre-apply is a separate step.

BLOCKER 2 — AND THIS ONE IS ARMED. Prod's external-dns Deployment runs
--policy=sync --registry=txt --domain-filter=keithjackson.dev WITH NO
--txt-owner-id. Absent that flag it defaults to `default` — the owner of ALL 36
of dev's TXT registry entries. Scaling it to 1 would claim dev's records and,
under policy: sync, DELETE THEM: the 2026-09-04 incident, exactly.

The repo is not at fault. helmrelease.yaml sets txtOwnerId from cluster-vars,
prod resolves it to `prod`, and the LIVE HelmRelease spec.values carries
txtOwnerId: "prod". Verified against the real chart: 1.21.1 with txtOwnerId:
prod renders --txt-owner-id=prod, so the key is right and an upgrade would fix
it. The fault is Helm/Flux STATE: `helm get values` shows the installed release
has policy and domainFilters but NO txtOwnerId — it predates c8c1d29 — while
helm-controller reports Ready=True, observedGeneration=3, and a history entry
whose configDigest EQUALS lastAttemptedConfigDigest. It believes it already
deployed the current config, so it will never upgrade on its own.

The only thing that prevented the incident was replicas=0 — manual drift from
the incident recovery, which any Helm upgrade resets to the chart default of 1.
That is not a guard, it is luck.

CONTAINED: `flux suspend helmrelease external-dns`, replicas left at 0. Costs
nothing (prod publishes no hostnames yet; app tiers suspended) and is reversible.
Remediation ordering matters and is documented: the naive resume + reconcile
--force adds the flag and starts the pod in ONE operation, leaving no window to
verify — a race whose failure mode is deleting dev's DNS. Pin replicas at 0,
confirm --txt-owner-id=prod in the live args, THEN scale.

Dev verified unharmed throughout: Cloudflare snapshotted before the unsuspend
and re-queried after — 57 records byte-for-byte identical, 36 TXT still
owner=default. Queried against the Cloudflare API, never a local resolver.

Refs R-GITOPS-01, R-ENV-02, R-SEC-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-04 23:54:33 -05:00
jakman85andClaude Opus 5 6b88670129 docs(prod): D0/D1 as executed, and CORRECT the P1 gate — it guards the apex, not prod
Two runbook corrections from executing step A of the dev migration.

1. P1 said "only NOW unsuspend prod", gated on D8 (dev's apex vacated). That
   silently assumed prod would be serving the bare apex. IT IS NOT. Prod runs
   c8c1d29, where BASE_DOMAIN=prod.home.keithjackson.dev and
   EXTERNALDNS_TXT_OWNER_ID=prod — verified live in prod's applied cluster-vars.
   Prod's names are a DISJOINT SUBTREE under a DISTINCT registry owner, so
   neither §10a incident mode applies: it cannot claim dev's `default`-owned
   records, and *.prod.home.keithjackson.dev is a DIFFERENT name set from dev's,
   so it has its own Let's Encrypt duplicate-cert bucket and races no
   _acme-challenge with dev. Prod can therefore be brought up IN FULL while dev
   still holds the apex — and should be, because it proves the whole stack
   before the flip, turning the cutover into a DNS/TLS change on a known-good
   cluster rather than a bring-up and a cutover simultaneously. D8 gates ONLY
   the later move of prod's BASE_DOMAIN to the apex.

2. D1 was executed via the REPOINT option, not suspend — repointing removes the
   dependency that causes the strand instead of merely deferring it. Dev now
   fetches from http://192.168.2.132:3000/... branch `dev-live`: IP-addressed,
   plain HTTP, dependent on NEITHER DNS NOR TLS, which is the property a DNS/TLS
   migration needs. A merge to `main` is now inert for dev.

   THE TRAP, recorded because it is not obvious: a bare copy of main@4809199 on
   the bootstrap Gitea would NOT have held. The root renders ./clusters/dev, and
   clusters/dev/kustomization.yaml CONTAINS the patch that sets
   GitRepository.spec.url — so Flux would have fetched from the bootstrap Gitea
   and then applied manifests that set the url straight back, reverting the
   repoint and re-arming the strand. The branch must declare its own source.
   dev-live (a4a3c2e) carries exactly one content change on 4809199, proven by
   `kubectl kustomize clusters/dev` differing in EXACTLY TWO of 357 lines.

   NEW DEPENDENCY, and it is the weak link: 192.168.2.132 is a DHCP lease on the
   workstation's WIFI (wlp0s20f3, NetworkManager ipv4.method=auto — confirmed by
   inspection, not assumed). A lease change or the workstation going offline
   stalls dev's FETCHING; the cluster keeps serving and the fix is a kubectl
   patch. Hence: a static reservation is worth having, and the window should be
   kept short — the exit is D4. Pointing Flux at gitea.gitea.svc.cluster.local
   would drop the DNS, TLS and workstation dependencies at once and is the better
   long-term shape, but namespace `gitea` has default-deny plus a Cilium policy
   admitting only the Gateway, so it needs a new authored policy — deliberately,
   not mid-migration.

Post-cutover verification: 13/13 Kustomizations Ready on dev-live@a4a3c2e,
cluster-vars byte-identical to the D0 snapshot (BASE_DOMAIN still the apex),
18/18 Longhorn volumes attached+healthy, all 16 HTTPRoute hostnames unchanged,
and the Cloudflare record set re-queried and diffed byte-for-byte against D0 —
identical, 57 records, 36 TXT all owner=default.

Refs R-ENV-02, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-04 23:39:02 -05:00
jakman85andClaude Opus 5 13081a863f feat(bootstrap): automate the Kyverno CRD pre-apply — the last cold-start blocker
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.

The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.

New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.

NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.

Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.

THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
  1. any derived coordinate that is empty/null/multi-valued aborts;
  2. rendered CRD count must be > 0 — this caught a real bug in the first
     draft: on an empty render `yq .metadata.name` emits one BLANK line, which
     counted as 1 CRD and sailed past the guard into an apply of an empty file
     returning 0, i.e. the exact silent-success trap. Blank/null lines are now
     dropped and a chart with no CRDs exits 1;
  3. every applied CRD is waited on for condition=Established, so "kubectl
     returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.

`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.

MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.

NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-04 23:32:10 -05:00
jakman85andClaude Opus 5 9137bbd0c3 feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex
*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***

Operator simplified the architecture, inverting c8c1d29:
  dev  -> *.dev.home.keithjackson.dev        (was the apex)
  prod -> *.home.keithjackson.dev            (the apex; no prod. subdomain, no CNAME layer)

WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.

Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.

VALUES
                BASE_DOMAIN                  TXT_OWNER_ID   DOMAIN_FILTER
  base          dev.home.keithjackson.dev    sentinel       keithjackson.dev
  dev           dev.home.keithjackson.dev    dev            keithjackson.dev
  prod          home.keithjackson.dev        prod           keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.

dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.

DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.

VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.

THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.

Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.

Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.

RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.

ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.

The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.

Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 23:08:50 -05:00
289 changed files with 4696 additions and 37614 deletions

No files matched your search

+20 -43
View File
@@ -1,44 +1,25 @@
# .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
#
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
# below for why the script, not `make`, is invoked here). It runs every stage in order —
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
# R-TEST-03 decision, printing
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
# single required status check on `main` (see branch protection). A non-zero exit fails the job,
# so a BLOCKED gate blocks the merge.
#
# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
# environment it is:
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
# is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
# `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
# says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
# REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
# spec-immutability pre-step below calls the script directly.
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
# the merge.
#
# The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
# no broad kubeconfig) driven by the separate `conformance` job below.
#
# NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
# early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
# re-runs it as its first stage, which is the authoritative one. The gate step below is the
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
# is retained dormant in merge-gate.yaml.
---
@@ -88,17 +69,13 @@ jobs:
- name: Spec immutability (R-TEST-06, skips if spec repo absent)
run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
# Aggregate merge gate: the same script a human runs in the devcontainer via
# `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
# render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
# job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
env:
GATE_STRICT_SKIPS: "true"
run: ./ci/gate.sh || [ "$?" -eq 3 ]
# Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
# (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
# render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
# BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
# CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
- name: Merge gate — make -C ci gate
run: make -C ci gate
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
# ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which
+2 -17
View File
@@ -39,21 +39,6 @@ install_chainsaw() {
curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
}
# promtool — the alert-rule UNIT-TEST engine the L0 check
# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
install_promtool() {
have promtool && return 0
log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
| tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
chmod +x "${bindir}/promtool"
}
install_kubectl() {
have kubectl && return 0
log "kubectl ${KUBECTL_VERSION}"
@@ -65,10 +50,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
case "$stage" in
lint) install_yamllint ;;
l0) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;; # schema (check-jsonschema) + Kyverno L0 + alert unit tests
l0) install_yamllint; install_check_jsonschema; install_kyverno ;; # schema (check-jsonschema) + Kyverno L0
l1) install_chainsaw ;; # L1 render via conformance runner (Chainsaw)
conformance) install_chainsaw; install_kubectl ;; # trigger the in-cluster Job
all) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
all) install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
*) echo "unknown stage: $stage" >&2; exit 2 ;;
esac
log "stage '${stage}' tools ready"
+3
View File
@@ -44,3 +44,6 @@ infra/tofu/.r2.env
# Vendored ansible collections (installed by ansible-galaxy for node re-provision; not tracked or linted)
infra/ansible/collections/
# standup.sh DNS snapshots — run artifacts, not authored state
.standup-dns/
@@ -1,97 +0,0 @@
# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
#
# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
# merely left alone. That is a one-way, outward-facing operation against a live authoritative
# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
# reconcile removes can be restored by hand from this file. It is the same discipline as
# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
# change rather than to a migration.
#
# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
# devcontainer, which has previously produced false "no records" readings — the API is the only
# reading that is both complete and authoritative.
#
# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
# TTL and the proxy flag, not just the value.
#
# <type>|<name>|<content>|<ttl>|<proxied>
#
# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
#
# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
# cannot touch them regardless of what this branch does:
#
# *.keithjackson.dev -> 136.51.115.252 legacy stack (the zone wildcard)
# brickbreak.keithjackson.dev -> 20.236.242.14 unrelated, off-prem
# dev.brickbreak… -> 20.236.242.14 unrelated, off-prem
#
# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
# end state for `internal` services and it is also precisely why Pi-hole must already be
# answering the LAN authoritatively before this branch merges.
#
# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
# HTTPRoute that already had a published record produced
# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
# interval (~60s) later, and nothing else in the zone moved.
#
# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
# points the registry at a source object that does not exist.
A|*.keithjackson.dev|136.51.115.252|1|false
A|auth.home.keithjackson.dev|192.168.2.20|1|false
A|brickbreak.keithjackson.dev|20.236.242.14|1|false
A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
A|crafty.home.keithjackson.dev|192.168.2.20|1|false
A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
A|gitea.home.keithjackson.dev|192.168.2.20|1|false
A|grafana.home.keithjackson.dev|192.168.2.20|1|false
A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
A|home.keithjackson.dev|192.168.2.20|1|false
A|jackett.home.keithjackson.dev|192.168.2.20|1|false
A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
A|pihole.home.keithjackson.dev|192.168.2.20|1|false
A|plex.home.keithjackson.dev|192.168.2.20|1|false
A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
A|radarr.home.keithjackson.dev|192.168.2.20|1|false
A|seerr.home.keithjackson.dev|192.168.2.20|1|false
A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
A|unifi.home.keithjackson.dev|192.168.2.21|1|false
TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false
-57
View File
@@ -1,57 +0,0 @@
A|*.keithjackson.dev|136.51.115.252
A|auth.home.keithjackson.dev|192.168.1.53
A|brickbreak.keithjackson.dev|20.236.242.14
A|crafty-mc.home.keithjackson.dev|192.168.1.55
A|crafty.home.keithjackson.dev|192.168.1.53
A|dev.brickbreak.keithjackson.dev|20.236.242.14
A|gitea.home.keithjackson.dev|192.168.1.53
A|grafana.home.keithjackson.dev|192.168.1.53
A|headlamp.home.keithjackson.dev|192.168.1.53
A|home.keithjackson.dev|192.168.1.53
A|jackett.home.keithjackson.dev|192.168.1.53
A|keycloak.home.keithjackson.dev|192.168.1.53
A|nextcloud.home.keithjackson.dev|192.168.1.53
A|pihole.home.keithjackson.dev|192.168.1.53
A|plex.home.keithjackson.dev|192.168.1.53
A|qbittorrent.home.keithjackson.dev|192.168.1.53
A|radarr.home.keithjackson.dev|192.168.1.53
A|seerr.home.keithjackson.dev|192.168.1.53
A|semaphore.home.keithjackson.dev|192.168.1.53
A|sonarr.home.keithjackson.dev|192.168.1.53
A|unifi.home.keithjackson.dev|192.168.1.54
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
-95
View File
@@ -1,95 +0,0 @@
A|*.keithjackson.dev|136.51.115.252
A|auth.home.keithjackson.dev|192.168.1.53
A|auth.prod.home.keithjackson.dev|192.168.2.20
A|brickbreak.keithjackson.dev|20.236.242.14
A|conformance-dns.home.keithjackson.dev|192.168.1.56
A|crafty-mc.home.keithjackson.dev|192.168.1.55
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
A|crafty.home.keithjackson.dev|192.168.1.53
A|crafty.prod.home.keithjackson.dev|192.168.2.20
A|dev.brickbreak.keithjackson.dev|20.236.242.14
A|gitea.home.keithjackson.dev|192.168.1.53
A|gitea.prod.home.keithjackson.dev|192.168.2.20
A|grafana.home.keithjackson.dev|192.168.1.53
A|grafana.prod.home.keithjackson.dev|192.168.2.20
A|headlamp.home.keithjackson.dev|192.168.1.53
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
A|home.keithjackson.dev|192.168.1.53
A|jackett.home.keithjackson.dev|192.168.1.53
A|jackett.prod.home.keithjackson.dev|192.168.2.20
A|keycloak.home.keithjackson.dev|192.168.1.53
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
A|nextcloud.home.keithjackson.dev|192.168.1.53
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
A|pihole.home.keithjackson.dev|192.168.1.53
A|pihole.prod.home.keithjackson.dev|192.168.2.20
A|plex.home.keithjackson.dev|192.168.1.53
A|plex.prod.home.keithjackson.dev|192.168.2.20
A|prod.home.keithjackson.dev|192.168.2.20
A|qbittorrent.home.keithjackson.dev|192.168.1.53
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
A|radarr.home.keithjackson.dev|192.168.1.53
A|radarr.prod.home.keithjackson.dev|192.168.2.20
A|seerr.home.keithjackson.dev|192.168.1.53
A|seerr.prod.home.keithjackson.dev|192.168.2.20
A|semaphore.home.keithjackson.dev|192.168.1.53
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
A|sonarr.home.keithjackson.dev|192.168.1.53
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
A|unifi.home.keithjackson.dev|192.168.1.54
A|unifi.prod.home.keithjackson.dev|192.168.2.21
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-conformance-dns.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/chainsaw-growing-grouse/dns-probe-svc
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
-93
View File
@@ -1,93 +0,0 @@
A|*.keithjackson.dev|136.51.115.252
A|auth.home.keithjackson.dev|192.168.1.53
A|auth.prod.home.keithjackson.dev|192.168.2.20
A|brickbreak.keithjackson.dev|20.236.242.14
A|crafty-mc.home.keithjackson.dev|192.168.1.55
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
A|crafty.home.keithjackson.dev|192.168.1.53
A|crafty.prod.home.keithjackson.dev|192.168.2.20
A|dev.brickbreak.keithjackson.dev|20.236.242.14
A|gitea.home.keithjackson.dev|192.168.1.53
A|gitea.prod.home.keithjackson.dev|192.168.2.20
A|grafana.home.keithjackson.dev|192.168.1.53
A|grafana.prod.home.keithjackson.dev|192.168.2.20
A|headlamp.home.keithjackson.dev|192.168.1.53
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
A|home.keithjackson.dev|192.168.1.53
A|jackett.home.keithjackson.dev|192.168.1.53
A|jackett.prod.home.keithjackson.dev|192.168.2.20
A|keycloak.home.keithjackson.dev|192.168.1.53
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
A|nextcloud.home.keithjackson.dev|192.168.1.53
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
A|pihole.home.keithjackson.dev|192.168.1.53
A|pihole.prod.home.keithjackson.dev|192.168.2.20
A|plex.home.keithjackson.dev|192.168.1.53
A|plex.prod.home.keithjackson.dev|192.168.2.20
A|prod.home.keithjackson.dev|192.168.2.20
A|qbittorrent.home.keithjackson.dev|192.168.1.53
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
A|radarr.home.keithjackson.dev|192.168.1.53
A|radarr.prod.home.keithjackson.dev|192.168.2.20
A|seerr.home.keithjackson.dev|192.168.1.53
A|seerr.prod.home.keithjackson.dev|192.168.2.20
A|semaphore.home.keithjackson.dev|192.168.1.53
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
A|sonarr.home.keithjackson.dev|192.168.1.53
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
A|unifi.home.keithjackson.dev|192.168.1.54
A|unifi.prod.home.keithjackson.dev|192.168.2.21
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
-117
View File
@@ -1,117 +0,0 @@
# CLAUDE.md — how work lands in this repo
Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
## 1. Never push to `main`. Ever.
Every durable change reaches `main` exactly one way:
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
There is no other path. Specifically:
- **Do not `git push origin main`.**
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
- **Do not merge your own PR.** A human approves and clicks merge.
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
> That wording is **stale** — it predates the PR flow. It describes what Flux does
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
| Approval required before merge | **`required_approvals: 0`** |
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
"required status check" with a runner online; neither was true, which is how this went
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
protection half is not — see `infra/gitea/`.
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 3. Gate before you push
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
Exit codes are a contract — do not flatten them:
| rc | Meaning |
|---|---|
| 0 | PASS |
| 1 | BLOCKED — something failed |
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
beside this one; that single skip is expected and tolerated. Any *other* skip is a
real gap in your run.
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
### Running the gate from a git worktree
Two unrelated failures, both with known workarounds — put both in any agent brief:
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
and exists only in the main checkout. Do **not** copy it (it is a credential).
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
normally.
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
the `ENGINE` hook — a shim that intercepts `run` and re-execs
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
`tracked_files()` is this bug, not a broken tree.
## 4. Branch naming
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
## 5. Cluster access
`kubectl` does not work on the operator workstation; the toolchain lives only in the
pinned container. Always name the context explicitly:
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
default has already changed once. Never rely on it. Do not run two `dev.sh`
invocations concurrently.
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
## 6. Secrets
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
list key names or object names only. A password has been leaked into a transcript
this way before.
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
the single creation-rules file; do not add a second one in a subtree.
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
flow. Do not design around committing one — read the expiry from the issuing page and
treat a burnt token as free to retry.
## 7. Before you finish
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
verified, and what you did **not** verify. State gaps plainly — an honest
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
burned by the difference.
+268 -3015
View File
File diff suppressed because it is too large. Load diff
+19 -42
View File
@@ -38,56 +38,22 @@ ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render
## Status
Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
hardware rather than on Proxmox VMs.
**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
alerts live*:
| criterion | state |
|---|---|
| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
exercise rather than something to run in passing.
Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
open** — `clusters/prod` is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
`./conformance/run.sh --coverage-only`; the full report is
`conformance/_artifacts/coverage/coverage.md`.
Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
second.
**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
the full report is `conformance/_artifacts/coverage/coverage.md`.
The running narrative — decisions, root causes, incidents, follow-ups — is
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
@@ -103,6 +69,17 @@ infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
```
`kubectl`/`flux`/`helm` in there talk to the **dev** cluster by default. Prefix `CLUSTER=prod` to
target prod instead:
```bash
infra/devcontainer/dev.sh run kubectl get nodes # dev (default)
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
```
Both clusters are always present in the container's `KUBECONFIG` as the contexts `k3s-dev` and
`k3s-prod`; `CLUSTER` picks which one is current, and each run prints the cluster it is targeting.
The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
[`infra/devcontainer/README.md`](infra/devcontainer/README.md).
+26 -147
View File
@@ -41,41 +41,10 @@ spec:
# disk KubeVirt cannot migrate and eviction would hard-stop the guest.
evictionStrategy: LiveMigrate
domain:
# =====================================================================================
# FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
# =====================================================================================
# MEASURED on prod 2026-09-14, not assumed:
# * the NoCloud seed KubeVirt builds for this VM carries
# {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
# (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
# EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
# had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
# was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
# standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
# * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
# cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
#
# CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
# worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
# unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
# every per-instance module (write_files, runcmd). The new payload is silently ignored.
#
# So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
# process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
# it also makes a rebuild reproducible — a fresh apply would otherwise take a random
# webhook-generated UUID and give the guest a different identity every time.
#
# The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
# difference is what makes the next boot a NEW cloud-init instance and installs the
# `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
# already stamped, so this change does not also churn the guest's DMI serial.
firmware:
uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
# FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
# so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
# is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
# the guest reports Ubuntu 24.04.4 LTS.
# is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
# add: firmware: { bootloader: { efi: { secureBoot: false } } }
cpu:
cores: 2
memory:
@@ -89,28 +58,9 @@ spec:
- name: cloudinit
disk:
bus: virtio
# The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
# iso9660 disk — see the `uosdata` volume near the bottom of this file.
- name: uosdata
disk:
bus: virtio
interfaces:
# MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
# KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
# /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
# that file while the NoCloud instance-id is unchanged — and the instance-id here is
# domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
# handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
# administratively DOWN with no address and no DHCP attempt.
# That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
# device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
# (because the per-instance modules were skipped), yet UOS is installed and
# qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
# Pinning the MAC to the value the current VMI already holds makes the guest's netplan
# match permanently, so this cannot recur on the next recreation.
- name: default
masquerade: {}
macAddress: a6:d7:c0:b3:72:fa
# Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
# these and (b) surfaces them as containerPorts on the virt-launcher pod so the
# LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
@@ -133,112 +83,41 @@ spec:
- name: rootdisk
dataVolume:
name: unifi-rootdisk
# =====================================================================================
# THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
# =====================================================================================
# KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
# each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
# reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
# REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
# URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
# operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
#
# `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
# Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
# and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
# ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
# guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
# only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
# works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
# read-only console output with no virtctl at all.
#
# WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
# 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
# concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
# L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
# envsubst over the DECRYPTED resources, so the escape contract in
# clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
# shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
# ONLY the operator's two values behind SOPS keeps both properties.
#
# A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
# disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
# re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
# the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
- name: uosdata
secret:
secretName: unifi-uos-installer
volumeLabel: UOSDATA
- name: cloudinit
cloudInitNoCloud:
# INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
# DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
# otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
# "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
# larger data." The payload below is ~1.5 KB; keep additions small.
#
# WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
# also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
# plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
# makes a later URL change take effect, because cloud-init will not re-run per-instance
# modules while the instance-id is unchanged.
#
# `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
# out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
# the serial console log for the current boot shows both `uosserver.service` and
# `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
# UOS is installed here; see the README for what is actually wrong (the guest lost its
# IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
# where the unit name changes in some future UOS release.
#
# ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
# 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
# is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
# which envsubst leaves untouched — verified against the live VMI, whose rendered
# userData still carries its shell variables verbatim. There is deliberately NO
# brace-form shell expansion left in this payload, so nothing here needs escaping; if
# you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
# write an indexed expansion.
# INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
# userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
# signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
# Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
# NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
# ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
# Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
# as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
# by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
# an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
# and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
# flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
# text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
# wording in clusters/base/apps-vms.yaml overstated it.
userData: |-
#cloud-config
hostname: unifi
fqdn: unifi.${BASE_DOMAIN}
package_update: true
package_upgrade: false
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
write_files:
- path: /usr/local/sbin/install-uos
permissions: '0755'
content: |
#!/bin/bash
#!/usr/bin/env bash
set -euo pipefail
d=/run/uos; mkdir -p $d
mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
p=$(tr -d '\r\n' < $d/console-password || true)
case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
s=/var/lib/uos-installed
[ -e $s ] && exit 0
u=$(tr -d '\r\n' < $d/installer-url || true)
case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
t=$(mktemp)
curl -fL --retry 5 --retry-delay 10 -o $t "$u"
chmod +x $t
yes y | $t install || yes y | $t
rm -f $t; date -Is > $s
- path: /etc/systemd/system/uos-install.service
content: |
[Unit]
Description=Install UniFi OS Server
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/install-uos
StandardOutput=journal+console
[Install]
WantedBy=multi-user.target
if [ -z "$${UOS_URL:-}" ]; then
UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
fi
[ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
runcmd:
- [systemctl, enable, --now, qemu-guest-agent]
- [systemctl, enable, --now, podman.socket]
- [systemctl, enable, --now, uos-install.service]
- [ systemctl, enable, --now, qemu-guest-agent ]
- [ systemctl, enable, --now, podman.socket ]
- [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+4 -57
View File
@@ -21,61 +21,8 @@
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
#
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
# into the guest.
#
# -------------------------------------------------------------------------------------------
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
# -------------------------------------------------------------------------------------------
# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
#
# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
# announcing node that holds no backend has nowhere to send the packet and answers ICMP port
# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
# (the leases were 8 days old); the operator's controller UI and device inform path were down.
# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
# deleting the Lease forces a re-election — and that re-election is random, so it can pick a
# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
# reference implementation this file now follows.
#
# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
# controller has always seen the pod-side masquerade gateway as the source, never the LAN
# client. There is a SECOND translation below the Service, and it already destroyed the very
# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
# it is giving up an address the guest could not see in the first place.
#
# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
# MAC (and the device reports its own IP inside the inform payload), not the packet source,
# and guest-portal authorization runs on the gateway appliance, not on this controller.
#
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
# is what makes this survive a reschedule and a rebuild.
#
# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
#
# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
# fix would be a NEW restriction that could silently break a remote site's device informing over
# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
# change, made deliberately and tested against the device fleet.
# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
# VIP pinned to the node running the VM (also what L2 announcement does).
#
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -102,7 +49,7 @@ metadata:
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
externalTrafficPolicy: Local
selector:
kubevirt.io/vm: unifi
ports:
@@ -125,7 +72,7 @@ metadata:
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
externalTrafficPolicy: Local
selector:
kubevirt.io/vm: unifi
ports:
+74 -202
View File
@@ -1,48 +1,18 @@
# UniFi Network controller (UniFi OS Server) — VM onboarding
> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
> cannot stall the app tier.
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
> or install cannot stall the app tier.
>
> **Status (re-verified against live prod 2026-09-14):**
> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
> - CDI import — **Succeeded** (100%).
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
> `:8080` refuse connections.
## What is actually wrong (2026-09-14)
The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
That was wrong, and acting on it would have meant reinstalling over a working controller.
Evidence, all read-only, from the live cluster:
| Claim | How it was checked | Result |
|---|---|---|
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
So the controller is installed and its units start; nothing downstream of the guest's own network
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
guest.** Diagnosing it needs a shell, which is the second problem below.
**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
new Secret is for.
> **Status (verified 2026-08-26):**
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
## What this is
@@ -59,39 +29,24 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
version. Browser access is via the L4 VIP. Re-add it if/when that lands.
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
### Cloud-init delivery — two constraints learned the hard way
1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
validate as an unknown field.
2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
nothing.** The NoCloud seed's `instance-id` is exactly
`spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
(so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
`runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
to force a re-run. See the long comment in `30-virtualmachine.yaml`.
3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
envsubst therefore runs over the cloud-init payload, and the escape contract in
`clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
indexed expansion is a hard error that fails the whole Kustomization.
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
Secret deliberately.
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
### Node prerequisites this VM forced onto the cluster
@@ -101,31 +56,15 @@ version. Browser access is via the L4 VIP. Re-add it if/when that lands.
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
3-replica class made the import fail with insufficient storage.
## UOS install method (researched 2026-09-14)
## UOS install method (researched)
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
(the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
Podman must be at `/usr/bin/podman`.
- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
- **Is the URL a short-lived signed link?** *Probably not.* The links look like
`https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
**no signature query string**, and public community installers hard-code them. It is
account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
account-derived material out of plaintext config — but do **not** build logic around expiry.
- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
ecosystem actually does is pipe confirmations in: the widely used community installer
(`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
`yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
(no URL has been supplied, and UOS is already installed here), so treat it as "established by
convergent third-party practice", not as verified-by-us.
- The previously shipped helper had two defects worth recording: it resolved the URL from
`fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
`unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
`ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
**`https://<host>:11443`**.
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
@@ -139,143 +78,76 @@ version. Browser access is via the L4 VIP. Re-add it if/when that lands.
| 3478 | UDP | **STUN** (device ⇄ controller) |
| 10001 | UDP | **L2 device discovery** |
**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
## L4 exposure design (the hard part)
The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
(`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
`default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
/ `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
declared ports into the guest.
- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
- **Device-adoption story:** point devices' **inform host** at the VIP
(`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
`set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
— via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
`minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover.
## The operator's one field — installing UOS without a shell
`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
`uos-install.service` reads it **on every boot**:
```bash
sops apps/vms/unifi/unifi-uos-installer.sops.yaml # fill installer-url and/or console-password
git commit && git push # merge to the branch prod's Flux tracks
kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
-c guest-console-log # watch the boot + the install, read-only
```
Design notes that matter if you change this:
- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
operator values in SOPS.
- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
firmware-UUID bump.
- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
controller on next boot".
- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
`apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
`clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
## Open questions / risks
1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
requires a shell, which requires `console-password` above plus one restart. First things to run
on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
`ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
capture the state before rebooting a second time if you can.
2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
install-method section: no official silent flag exists, `yes y | <installer> install` is the
convergent community practice, and it has not been executed on our guest. Treat the first real
install as something to watch on the console log, not as a fire-and-forget.
3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
(`.23`) or the node addresses (`.10/.11/.12`).
6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
installer and for optional remote access.
8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
`kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
`virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
the cluster's **v1.9.0**.
## Remaining bring-up
Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
is written for a from-scratch cluster; the prod addresses are the live ones.
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
Kustomization (not an app wave).
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
`192.168.2.21`.
5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
`uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
fill `installer-url`, commit, and the guest installs itself on first boot.
6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
`192.168.1.54`.
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
complete the first-run wizard.
8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
split-horizon) and from in-cluster (coredns-custom).
9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
connects (green). Test guest portal / throughput if used.
10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
**Quick health check today:**
```bash
kubectl -n unifi get dv,vmi,svc
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
# the two that would have caught the current failure:
kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}' # expect an ipAddress
kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
-c compute -- ip neigh # expect 10.0.2.2 REACHABLE
```
-8
View File
@@ -11,11 +11,3 @@ resources:
- 30-virtualmachine.yaml
- 40-networkpolicy.yaml
- 50-service-l4.yaml
# SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
# into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
# Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
# FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
# template + the operator hand-off steps live in the `.example` sibling of the file below, which
# is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
# text contains an example template's filename, on the assumption it is being applied.
- unifi-uos-installer.sops.yaml
@@ -1,29 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: unifi-uos-installer
namespace: unifi
type: Opaque
stringData:
installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-19T02:47:13Z"
mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
@@ -1,71 +0,0 @@
# =============================================================================
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
#
# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
#
# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
# working controller. `console-password` IS useful today: it is the only way to get a shell.
#
# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
# ours to redistribute. Treat it as account-derived material, not as public configuration.
# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
# and public community install scripts hard-code these URLs — so it does not behave like a
# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
#
# HAND-OFF STEPS (operator) — this is the whole job:
# (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
# copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
# fetches it itself.
# (b) Open the encrypted sibling in your editor:
# sops apps/vms/unifi/unifi-uos-installer.sops.yaml
# (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
# placeholder rather than failing.
# (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
# (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
# kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
# Watch it come up WITHOUT virtctl:
# kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
# -o name | head -1) -c guest-console-log
# (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
#
# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
# payload ITSELF changes.
#
# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
# =============================================================================
apiVersion: v1
kind: Secret
metadata:
name: unifi-uos-installer
namespace: unifi
type: Opaque
stringData:
# KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
# /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
#
# The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
# (this placeholder included) is treated by the guest as "not provided yet": the unit logs
# "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
installer-url: REPLACE-ME
# Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
# THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
# no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
# only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
# virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
# openssl rand -base64 24
# Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
console-password: REPLACE-ME
+22 -23
View File
@@ -170,27 +170,6 @@ spec:
- port: '3000'
protocol: TCP
---
# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-flux-ingress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-flux-ingress
namespace: gitea
labels:
app.kubernetes.io/name: gitea
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: gitea
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: flux-system
toPorts:
- ports:
- port: '3000'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
@@ -238,6 +217,26 @@ spec:
- name: gitea
port: 3000
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: gitea
namespace: gitea
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: gitea
spec:
selector:
matchLabels:
app.kubernetes.io/name: gitea
namespaceSelector:
matchNames:
- gitea
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -252,11 +251,11 @@ spec:
- name: gitea.slis
rules:
- alert: gitea-down
expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
expr: up{job="gitea"} == 0
for: 5m
labels:
severity: critical
service: gitea
annotations:
summary: gitea is down
description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'gitea has been unreachable for 5m (SLI: up).'
+25 -7
View File
@@ -74,7 +74,7 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
homelab.keithjackson.dev/auth-groups: media-users
spec:
containers:
- name: jackett
@@ -83,7 +83,7 @@ spec:
- containerPort: 9117
volumeMounts:
- name: config
mountPath: /config
mountPath: /data/config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -97,7 +97,6 @@ spec:
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
@@ -267,8 +266,7 @@ metadata:
app.kubernetes.io/name: jackett
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
external-dns.alpha.kubernetes.io/controller: none
homelab.keithjackson.dev/auth-groups: media-users
spec:
parentRefs:
- name: shared
@@ -285,6 +283,26 @@ spec:
- name: jackett
port: 4180
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: jackett
namespace: jackett
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: jackett
spec:
selector:
matchLabels:
app.kubernetes.io/name: jackett
namespaceSelector:
matchNames:
- jackett
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -299,11 +317,11 @@ spec:
- name: jackett.slis
rules:
- alert: jackett-down
expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
expr: up{job="jackett"} == 0
for: 5m
labels:
severity: critical
service: jackett
annotations:
summary: jackett is down
description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'jackett has been unreachable for 5m (SLI: up).'
-5
View File
@@ -11,18 +11,13 @@ resources:
- jackett-oauth2-proxy-credentials.sops.yaml
- jackett.yaml
- nextcloud-db-app.sops.yaml
- nextcloud-db-backup-credentials-prod.sops.yaml
- nextcloud-db-backup-credentials.sops.yaml
- nextcloud-db.yaml
- pihole-dns.yaml
- pihole-oauth2-proxy-credentials.sops.yaml
- pihole.yaml
- plex-claim-secret.sops.yaml
- plex.yaml
- qbittorrent-bootstrap-secret.sops.yaml
- qbittorrent-oauth2-proxy-credentials.sops.yaml
- qbittorrent.yaml
- semaphore-db-app.sops.yaml
- semaphore-db-backup-credentials-prod.sops.yaml
- semaphore-db-backup-credentials.sops.yaml
- semaphore-db.yaml
@@ -1,29 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: nextcloud-db-backup-credentials-prod
namespace: nextcloud-db
type: Opaque
stringData:
ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-07T06:36:33Z"
mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+17 -303
View File
@@ -37,53 +37,16 @@ metadata:
name: nextcloud-db
namespace: nextcloud-db
spec:
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
# platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
# replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
# primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
# Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
# Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
# firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
#
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
instances: 3
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
# keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
# HA as a future hardening.
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
# Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
# nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
labels:
app.kubernetes.io/name: nextcloud-db
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
# back into the single-node failure domain this change removes. `required` makes it a constraint;
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
#
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
# switchover. Expect a single brief connection reset for Nextcloud.
affinity:
enablePodAntiAffinity: true
podAntiAffinityType: required
primaryUpdateMethod: switchover
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
# standby into a total write outage. `preferred` relaxes to async instead.
#
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
postgresql:
synchronous:
method: any
number: 1
dataDurability: preferred
storage:
storageClass: longhorn
size: 5Gi
@@ -105,38 +68,29 @@ spec:
memory: 256Mi
limits:
memory: 512Mi
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
# s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
# s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
# scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
# generated golden — render.py places the -prod sidecar in this DB's wave); the
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
# disk-full clock below rather than a merely missing backup.
#
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
# (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
# Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
backup:
retentionPolicy: "30d"
barmanObjectStore:
# PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
# NAME — and both environments have a `nextcloud-db`. Sharing one MinIO with a shared path
# would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
# destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
destinationPath: ${CNPG_BACKUP_PATH}
endpointURL: ${S3_BACKUP_ENDPOINT}
s3Credentials:
accessKeyId:
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
name: nextcloud-db-backup-credentials
key: ACCESS_KEY_ID
secretAccessKey:
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
name: nextcloud-db-backup-credentials
key: ACCESS_SECRET_KEY
wal:
compression: gzip
@@ -158,243 +112,3 @@ spec:
cluster:
name: nextcloud-db
method: barmanObjectStore
---
# =============================================================================
# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
#
# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
#
# nextcloud-db-1/-2/-3, every scrape:
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
# DETAIL: User does not have CONNECT privilege.
# msg="Error collecting user query" targetDatabase=nextcloud
#
# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
# while the scrape target still reported up=1 — a blind spot that looks like health.
# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
# been monitored.
#
# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
# their bootstrap; the difference is entirely in the live ACL:
#
# nextcloud : datacl = {nextcloud=CTc/nextcloud} <-- explicit, no PUBLIC entry
# keycloak : datacl = NULL
# semaphore : datacl = NULL
#
# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
# implicit CONNECT. On `nextcloud` something ran the equivalent of
# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
# git-declared fix is needed rather than another hand-run psql.
#
# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
# leave the exporter depending on an implicit default again — the precise fragility
# that broke here and that keycloak/semaphore are still relying on. Naming the
# exporter role makes the dependency explicit and survives a future REVOKE FROM
# PUBLIC on any of the three databases.
#
# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
# THIS. Checked against the live CRDs on prod:
# * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
# `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
# everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
# lists on `fdws` and `servers`.
# * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
# superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
# wrong tool: the only local role holding CONNECT here is the OWNER, so
# `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
# the database and read/write on every application table — the opposite of
# least privilege, to buy one CONNECT.
# * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
# repair these three already-bootstrapped instances (the actual bug), and on a
# FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
# would be a second mechanism that fixes nothing this one does not.
# So the SQL has to be applied by something, and this repo already has the pattern
# for state Flux cannot reach: the level-triggered CronJob reconciler in
# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
# silently reopen the blind spot. A loop converges instead.
#
# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
# GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
# write, no table or schema grants. Nothing further is needed because CNPG already
# creates this role as a member of the built-in `pg_monitor` role (verified on all
# three prod clusters), and pg_monitor already carries the read access every
# collector query wants — they read pg_catalog views (pg_stat_database,
# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
# went to ZERO on all three pods, i.e. every collector query now succeeds.
#
# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
# owner of a database holds the grant option on that database's privileges, so
# `nextcloud` can issue this GRANT itself. Verified live before committing, by
# running the statement under `SET ROLE nextcloud`. Credentials come from the
# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
#
# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
# header), so Kyverno generates no default-deny baseline here and this same-namespace
# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
#
# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
# make this reconciler a permitted destination for the Nextcloud app. It gets its own
# name so that selector keeps meaning "the database".
apiVersion: batch/v1
kind: CronJob
metadata:
name: nextcloud-db-exporter-grant
namespace: nextcloud-db
labels:
app.kubernetes.io/name: nextcloud-db-exporter-grant
app.kubernetes.io/part-of: nextcloud-db
spec:
# Every 15 minutes, matching the keycloak reconciler's cadence. A converged
# database makes this one idempotent GRANT and exits; the cost is negligible, and
# after a rebuild or PITR restore monitoring is blind for at most one interval
# (well inside the alert's `for: 6h`).
schedule: "*/15 * * * *"
# A run still waiting on the primary must not be lapped by the next tick.
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 300
ttlSecondsAfterFinished: 3600
template:
metadata:
labels:
app.kubernetes.io/name: nextcloud-db-exporter-grant
app.kubernetes.io/part-of: nextcloud-db
spec:
restartPolicy: Never
# Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
# no Kubernetes API access at all (R-SEC-02, least privilege).
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
# uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
runAsUser: 26
runAsGroup: 26
seccompProfile:
type: RuntimeDefault
containers:
- name: grant
# The SAME image the instances above run, pinned to the same tag. psql
# is already in it and it is already present on every node, so this
# adds no new image and no new supply-chain surface. Keep the tag in
# step with spec.imageName above when Renovate bumps it.
image: ghcr.io/cloudnative-pg/postgresql:16.4
imagePullPolicy: IfNotPresent
command:
- psql
# -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
# FAIL LOUDLY rather than reporting success on a refused grant.
- "-w"
- "-X"
- "-v"
- "ON_ERROR_STOP=1"
- "-f"
- /etc/grant/grant.sql
env:
# libpq connection, assembled from the owner Secret. Target the -rw
# Service: GRANT is a write and must reach the primary.
- name: PGHOST
value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
- name: PGPORT
value: "5432"
# Connecting to the application database itself is deliberate: the
# owner has CONNECT on it, and a successful connection is the same
# check the exporter was failing.
- name: PGDATABASE
value: nextcloud
- name: PGUSER
valueFrom:
secretKeyRef:
name: nextcloud-db-app
key: username
# Read by libpq from the environment and never logged; psql echoes
# no connection parameters.
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: nextcloud-db-app
key: password
- name: PGCONNECT_TIMEOUT
value: "10"
volumeMounts:
- name: sql
mountPath: /etc/grant
readOnly: true
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
volumes:
- name: sql
configMap:
name: nextcloud-db-exporter-grant
defaultMode: 0444
---
# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
# turns "the grant did not take" into a failed Job instead of a silent success.
apiVersion: v1
kind: ConfigMap
metadata:
name: nextcloud-db-exporter-grant
namespace: nextcloud-db
labels:
app.kubernetes.io/name: nextcloud-db-exporter-grant
app.kubernetes.io/part-of: nextcloud-db
data:
grant.sql: |
-- The ONE privilege the CNPG metrics exporter is missing on this database.
-- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
-- Read access for the collector's pg_catalog queries already comes from the
-- built-in pg_monitor role, which CNPG grants this role when it creates it.
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
-- Fail the Job if the privilege is somehow still absent, so a broken
-- reconciler surfaces as a failed Job rather than as a quiet blind spot.
-- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
-- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
-- being shipped this way: the ConfigMap is rendered through Flux's postBuild
-- substitution, and envsubst collapses a doubled dollar into a single one. The block
-- reached Postgres with its opening quote halved and every Flux-reconciled run died
-- with a syntax error pointing at that character.
-- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
-- the defect was invisible until Flux owned the object, and three runs of Error pods
-- were the first sign. Escaping by doubling again would work and would also be the
-- next person's trap; plain SQL leaves substitution nothing to touch.
--
-- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
-- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
-- is not a constant, so it cannot be folded away at plan time.
\echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
AS exporter_can_connect;
-196
View File
@@ -1,196 +0,0 @@
---
# =================================================================================================
# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
# =================================================================================================
# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
# schema has no word for.
#
# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
#
# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
# can never be allowed to move.
#
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
# and confusingly.
#
# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
# filter below exists to refuse. Internal name, internal answer, no public record — which is also
# why this change touches nothing under platform/externaldns/.
#
# -------------------------------------------------------------------------------------------------
# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
# -------------------------------------------------------------------------------------------------
# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
# not assumed (Cilium 1.20.1, prod, 2026-09-14):
#
# * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
# CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
# NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
# landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
# `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
# * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
# packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
# nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
# * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
# rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
# minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
# re-election — and that re-election is random, so it may pick a backend-less node again.
#
# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
# answering.
#
# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
#
# -------------------------------------------------------------------------------------------------
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
# -------------------------------------------------------------------------------------------------
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
# range to a /24 that excludes the test client, after which the same query that had been answering
# stopped being answered at all, and restoring it, after which it answered again.
#
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
# arrives from the ingress node's internal router address and Cilium resolves that source to the
# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
apiVersion: v1
kind: Service
metadata:
name: pihole-dns-udp
namespace: pihole
labels:
app.kubernetes.io/name: pihole
annotations:
lbipam.cilium.io/sharing-key: "pihole-dns"
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
- { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
---
apiVersion: v1
kind: Service
metadata:
name: pihole-dns-tcp
namespace: pihole
labels:
app.kubernetes.io/name: pihole
annotations:
lbipam.cilium.io/sharing-key: "pihole-dns" # same key + same IP => shares the VIP with -udp
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
- { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
---
# =================================================================================================
# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
# =================================================================================================
# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
# it and no policy for it.
#
# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
#
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
#
# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
#
# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
# node-to-node forwarding that IS the Cluster-policy datapath both need them.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: pihole-allow-lan-dns-ingress
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: pihole
ingress:
- fromEntities:
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
- host # this node's kubelet probes
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
toPorts:
- ports:
- { port: "53", protocol: UDP }
- { port: "53", protocol: TCP }
+22 -12
View File
@@ -81,15 +81,6 @@ spec:
image: pihole/pihole
ports:
- containerPort: 80
env:
- name: FTLCONF_dns_listeningMode
value: ALL
- name: FTLCONF_dns_upstreams
value: 8.8.8.8;8.8.4.4
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
volumeMounts:
- name: config
mountPath: /data/config
@@ -232,7 +223,6 @@ metadata:
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: platform-admins
external-dns.alpha.kubernetes.io/controller: none
spec:
parentRefs:
- name: shared
@@ -249,6 +239,26 @@ spec:
- name: pihole
port: 4180
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: pihole
namespace: pihole
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: pihole
spec:
selector:
matchLabels:
app.kubernetes.io/name: pihole
namespaceSelector:
matchNames:
- pihole
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -263,11 +273,11 @@ spec:
- name: pihole.slis
rules:
- alert: pihole-down
expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
expr: up{job="pihole"} == 0
for: 5m
labels:
severity: critical
service: pihole
annotations:
summary: pihole is down
description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'pihole has been unreachable for 5m (SLI: up).'
-69
View File
@@ -1,69 +0,0 @@
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
#
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
# not prose, so the upstream conditions below are described in words.
#
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
#
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
# the same file extracted from the running `plexinc/pms-docker:latest` image):
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
# README: "If server is already logged in, this parameter is ignored."
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
# empty body. curl's exit status is never checked, the sed for the authentication-token element
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
# It does NOT crashloop.
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
# by hand at claim time.
#
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
# Never decrypt this file into a shell, a log, or a shared terminal.
apiVersion: v1
kind: Secret
metadata:
name: plex-claim
namespace: plex
labels:
app.kubernetes.io/name: plex
type: Opaque
stringData:
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-29T06:31:16Z"
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+22 -58
View File
@@ -35,34 +35,6 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume plex-media - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: plex-media
labels:
app.kubernetes.io/name: plex
spec:
capacity:
storage: 9Ti
accessModes:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
readOnly: true
claimRef:
namespace: plex
name: plex-media
---
# PVC plex-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -79,23 +51,6 @@ spec:
requests:
storage: 8Gi
---
# PVC plex-media (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: plex-media
namespace: plex
labels:
app.kubernetes.io/name: plex
spec:
accessModes:
- ReadOnlyMany
storageClassName: ''
volumeName: plex-media
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -132,24 +87,13 @@ spec:
value: Etc/UTC
- name: ALLOWED_NETWORKS
value: 10.0.0.0/8,192.168.0.0/16
- name: PLEX_CLAIM
valueFrom:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
volumeMounts:
- name: config
mountPath: /config
- name: media
mountPath: /media
readOnly: true
volumes:
- name: config
persistentVolumeClaim:
claimName: plex-config
- name: media
persistentVolumeClaim:
claimName: plex-media
---
# Service
apiVersion: v1
@@ -249,6 +193,26 @@ spec:
- name: plex
port: 32400
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: plex
namespace: plex
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: plex
spec:
selector:
matchLabels:
app.kubernetes.io/name: plex
namespaceSelector:
matchNames:
- plex
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -263,11 +227,11 @@ spec:
- name: plex.slis
rules:
- alert: plex-down
expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
expr: up{job="plex"} == 0
for: 5m
labels:
severity: critical
service: plex
annotations:
summary: plex is down
description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'plex has been unreachable for 5m (SLI: up).'
@@ -1,60 +0,0 @@
# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
#
# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
# saved download-client credentials stop working at the next qbittorrent pod restart, silently
# breaking Completed Download Handling; (b) a human operator has no stable login at all and must
# read the pod log after every restart.
#
# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
# credential, because the hook re-asserts it on every start rather than once.
#
# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
# the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
# entries store in their own databases (app state on their config PVCs, NOT in git — see
# DECISIONS-LOG), so rotating it here means re-pointing both arrs.
# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
# payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
#
# NEVER put a value in a comment: SOPS does NOT encrypt comments.
# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
apiVersion: v1
kind: Secret
metadata:
name: qbittorrent-bootstrap
namespace: qbittorrent
labels:
app.kubernetes.io/name: qbittorrent
type: Opaque
stringData:
webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-13T05:23:23Z"
mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+25 -83
View File
@@ -35,33 +35,6 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: qbittorrent-downloads
labels:
app.kubernetes.io/name: qbittorrent
spec:
capacity:
storage: 9Ti
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
claimRef:
namespace: qbittorrent
name: qbittorrent-downloads
---
# PVC qbittorrent-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -78,23 +51,6 @@ spec:
requests:
storage: 5Gi
---
# PVC qbittorrent-downloads (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: qbittorrent-downloads
namespace: qbittorrent
labels:
app.kubernetes.io/name: qbittorrent
spec:
accessModes:
- ReadWriteMany
storageClassName: ''
volumeName: qbittorrent-downloads
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -118,43 +74,16 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
homelab.keithjackson.dev/auth-groups: media-users
spec:
containers:
- name: qbittorrent
image: lscr.io/linuxserver/qbittorrent:5.1.2
ports:
- containerPort: 8080
env:
- name: PUID
value: '1000'
- name: PGID
value: '3000'
- name: QBT_WEBUI_USERNAME
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-username
- name: QBT_WEBUI_PASSWORD
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-password
volumeMounts:
- name: config
mountPath: /config
- name: downloads
mountPath: /downloads
subPath: downloads
lifecycle:
postStart:
exec:
command:
- /bin/bash
- -c
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
mountPath: /data/config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -168,7 +97,6 @@ spec:
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
@@ -220,15 +148,10 @@ spec:
port: http
initialDelaySeconds: 5
periodSeconds: 15
securityContext:
fsGroup: 3000
volumes:
- name: config
persistentVolumeClaim:
claimName: qbittorrent-config
- name: downloads
persistentVolumeClaim:
claimName: qbittorrent-downloads
---
# Service
apiVersion: v1
@@ -343,8 +266,7 @@ metadata:
app.kubernetes.io/name: qbittorrent
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
external-dns.alpha.kubernetes.io/controller: none
homelab.keithjackson.dev/auth-groups: media-users
spec:
parentRefs:
- name: shared
@@ -361,6 +283,26 @@ spec:
- name: qbittorrent
port: 4180
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: qbittorrent
namespace: qbittorrent
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: qbittorrent
spec:
selector:
matchLabels:
app.kubernetes.io/name: qbittorrent
namespaceSelector:
matchNames:
- qbittorrent
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -375,11 +317,11 @@ spec:
- name: qbittorrent.slis
rules:
- alert: qbittorrent-down
expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
expr: up{job="qbittorrent"} == 0
for: 5m
labels:
severity: critical
service: qbittorrent
annotations:
summary: qbittorrent is down
description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'qbittorrent has been unreachable for 5m (SLI: up).'
@@ -1,29 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: semaphore-db-backup-credentials-prod
namespace: semaphore-db
type: Opaque
stringData:
ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-07T06:36:33Z"
mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+17 -63
View File
@@ -38,53 +38,16 @@ metadata:
name: semaphore-db
namespace: semaphore-db
spec:
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
# nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
# Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
# `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
# down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
# the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
# with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
#
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
instances: 3
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
# keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
# Postgres-level HA as a future hardening.
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
labels:
app.kubernetes.io/name: semaphore-db
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
# back into the single-node failure domain this change removes. `required` makes it a constraint;
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
#
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
# switchover. Expect a single brief connection reset for Semaphore.
affinity:
enablePodAntiAffinity: true
podAntiAffinityType: required
primaryUpdateMethod: switchover
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
# standby into a total write outage. `preferred` relaxes to async instead.
#
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
postgresql:
synchronous:
method: any
number: 1
dataDurability: preferred
storage:
storageClass: longhorn
size: 5Gi
@@ -106,38 +69,29 @@ spec:
memory: 256Mi
limits:
memory: 512Mi
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
# s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
# s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
# scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
# generated golden — render.py places the -prod sidecar in this DB's wave); the
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
# disk-full clock below rather than a merely missing backup.
#
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
# (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
backup:
retentionPolicy: "30d"
barmanObjectStore:
# PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
# NAME — and both environments have a `semaphore-db`. Sharing one MinIO with a shared path
# would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
# destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
destinationPath: ${CNPG_BACKUP_PATH}
endpointURL: ${S3_BACKUP_ENDPOINT}
s3Credentials:
accessKeyId:
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
name: semaphore-db-backup-credentials
key: ACCESS_KEY_ID
secretAccessKey:
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
name: semaphore-db-backup-credentials
key: ACCESS_SECRET_KEY
wal:
compression: gzip
-267
View File
@@ -1,267 +0,0 @@
# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
# tests run against the exact expressions Flux applies, never a copy.
#
# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
# (the merge gate runs the same harness via
# conformance/tests/l0-static/alert-rules-unit-tested)
#
# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
# workloads (verified on prod), and every rule below is pinned in BOTH directions:
# * QUIET on the healthy state -> it is not permanently firing;
# * FIRING when the workload has no ready replica -> it is not inert;
# * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
# (a rule that goes quiet because its series vanished) is itself alerted on.
#
# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
# ===========================================================================================
# 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
# This is the "not permanently firing" half. It is also the test that catches a mis-spelled
# label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
# matches everything, so this test goes red instead of the rule going quietly wrong.
# ===========================================================================================
- interval: 1m
name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
input_series:
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
values: '1+0x20'
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: gitea-down
exp_alerts: []
- eval_time: 10m
alertname: jackett-down
exp_alerts: []
- eval_time: 10m
alertname: pihole-down
exp_alerts: []
- eval_time: 10m
alertname: plex-down
exp_alerts: []
- eval_time: 10m
alertname: qbittorrent-down
exp_alerts: []
- eval_time: 10m
alertname: vm-demo-down
exp_alerts: []
# ===========================================================================================
# 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
# The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
# would light up every app at once and be useless for routing.
# ===========================================================================================
- interval: 1m
name: wave-0 plex has no available replica - only plex-down fires
input_series:
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
values: '1+0x20'
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: plex-down
exp_alerts:
- exp_labels:
severity: critical
service: plex
namespace: plex
deployment: plex
exp_annotations:
summary: "plex is down"
description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: gitea-down
exp_alerts: []
- eval_time: 10m
alertname: jackett-down
exp_alerts: []
- eval_time: 10m
alertname: pihole-down
exp_alerts: []
- eval_time: 10m
alertname: qbittorrent-down
exp_alerts: []
- eval_time: 10m
alertname: vm-demo-down
exp_alerts: []
# Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
# a single scrape blip does not page anyone.
promql_expr_test:
- expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
eval_time: 2m
exp_samples:
- labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
value: 0
- interval: 1m
name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
input_series:
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
values: '0+0x20'
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: gitea-down
exp_alerts:
- exp_labels:
severity: critical
service: gitea
namespace: gitea
deployment: gitea
exp_annotations:
summary: "gitea is down"
description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: jackett-down
exp_alerts:
- exp_labels:
severity: critical
service: jackett
namespace: jackett
deployment: jackett
exp_annotations:
summary: "jackett is down"
description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: pihole-down
exp_alerts:
- exp_labels:
severity: critical
service: pihole
namespace: pihole
deployment: pihole
exp_annotations:
summary: "pihole is down"
description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: qbittorrent-down
exp_alerts:
- exp_labels:
severity: critical
service: qbittorrent
namespace: qbittorrent
deployment: qbittorrent
exp_annotations:
summary: "qbittorrent is down"
description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: plex-down
exp_alerts: []
- eval_time: 10m
alertname: vm-demo-down
exp_alerts: []
# ===========================================================================================
# 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
# The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
# the Deployment series for plex is simply not in the input at all, and plex-down must still
# fire (via the `absent()` half), with the labels the selector's equality matchers carry.
# ===========================================================================================
- interval: 1m
name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
input_series:
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
values: '1+0x20'
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: plex-down
exp_alerts:
- exp_labels:
severity: critical
service: plex
namespace: plex
deployment: plex
exp_annotations:
summary: "plex is down"
description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: gitea-down
exp_alerts: []
# ===========================================================================================
# 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
# series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
# some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
# is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
# (a capital-R matcher selects nothing and would be just as inert).
# ===========================================================================================
- interval: 1m
name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
input_series:
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: vm-demo-down
exp_alerts:
- exp_labels:
severity: critical
service: vm-demo
namespace: vm-demo
name: vm-demo
phase: running
exp_annotations:
summary: "vm-demo is down"
description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
- interval: 1m
name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
input_series:
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: vm-demo-down
exp_alerts:
- exp_labels:
severity: critical
service: vm-demo
namespace: vm-demo
name: vm-demo
phase: running
exp_annotations:
summary: "vm-demo is down"
description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
-139
View File
@@ -1,139 +0,0 @@
---
# Namespace (trust zone; default-deny + policies apply)
apiVersion: v1
kind: Namespace
metadata:
name: vm-demo
labels:
homelab.keithjackson.dev/app-namespace: 'true'
annotations:
homelab.keithjackson.dev/backup-policy: daily-offcluster
---
# Default-deny baseline (R-NET-01): deny all, allow only DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: vm-demo
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# CDI DataVolume (standalone; survives rebuild, R-STATE-01)
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: vm-demo-rootdisk
namespace: vm-demo
spec:
source:
registry:
url: docker://quay.io/containerdisks/fedora:41
storage:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 6Gi
---
# KubeVirt VirtualMachine (R-PLAT-03)
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-demo
namespace: vm-demo
labels:
app.kubernetes.io/name: vm-demo
spec:
running: false
template:
metadata:
labels:
kubevirt.io/vm: vm-demo
homelab.keithjackson.dev/egress-class: internet
homelab.keithjackson.dev/auth-mode: none
spec:
domain:
cpu:
cores: 1
memory:
guest: 1536Mi
devices:
disks:
- name: rootdisk
disk:
bus: virtio
- name: cloudinit
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 1536Mi
networks:
- name: default
pod: {}
volumes:
- name: rootdisk
dataVolume:
name: vm-demo-rootdisk
- name: cloudinit
cloudInitNoCloud:
userData: "#cloud-config\npackage_update: true\npackages:\n - dnf-automatic\nwrite_files:\n - path: /etc/dnf/automatic.conf\n content: |\n [commands]\n upgrade_type = default\n random_sleep = 0\n download_updates = yes\n apply_updates = yes\nruncmd:\n - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): vm-demo-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: vm-demo-allow-internet-egress
namespace: vm-demo
labels:
app.kubernetes.io/name: vm-demo
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: vm-demo
egress:
- toEntities:
- world
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: vm-demo-slis
namespace: vm-demo
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: vm-demo
spec:
groups:
- name: vm-demo.slis
rules:
- alert: vm-demo-down
expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
for: 5m
labels:
severity: critical
service: vm-demo
annotations:
summary: vm-demo is down
description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'
@@ -14,7 +14,7 @@ metadata:
app.kubernetes.io/name: act-runner
type: Opaque
stringData:
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
sops:
kms: []
gcp_kms: []
@@ -30,8 +30,8 @@ sops:
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-29T05:48:35Z"
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
lastmodified: "2026-08-26T04:48:14Z"
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+106 -61
View File
@@ -99,71 +99,70 @@ spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14).
automountServiceAccountToken: false
# ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
#
# NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
# `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
# running alongside them, and is terminated after them. kubelet holds `runner` until
# this container's startupProbe passes, so the ordering is enforced by the kubelet
# instead of hand-rolled shell in the runner.
#
# This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
# Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
# postBuild substitution over this ENTIRE manifest, comments included, and those loops
# depended on bare dollar-names surviving it untouched. There is now no shell here at
# all, so there is nothing to escape or mis-escape.
#
# WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
# re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
# and only THEN begins listening, so the port can accept a connection while the certs
# are absent or incomplete -- and act_runner hard-exits on a missing
# /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
# are usable and that the daemon answers. It also removes the one-per-start
# "TLS handshake error ... EOF" that the old `nc -z` probe logged in dockerd.
#
# periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
initContainers:
- name: dind
restartPolicy: Always # <- this field is what makes it a SIDECAR
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
startupProbe:
exec:
command:
- docker
- --host=tcp://127.0.0.1:2376
- --tlsverify
- --tlscacert=/certs/client/ca.pem
- --tlscert=/certs/client/cert.pem
- --tlskey=/certs/client/key.pem
- info
periodSeconds: 1
failureThreshold: 60
containers:
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
- name: runner
image: gitea/act_runner:0.2.11
# No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
# registers from env on first boot and runs `act_runner daemon`. The two wait-loops
# that used to live here are gone: kubelet now holds this container until the dind
# sidecar's startupProbe passes (see initContainers above).
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
# boot and runs `act_runner daemon`.
#
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
# and hard-exits if it is missing (a startup race against the sidecar).
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
#
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
# braced form or the double-dollar form; both are wrong here, for different reasons.
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
# cluster verbatim and both 60s bounds below are real, enforced bounds.
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
# form — still wrong for us, because on any path where envsubst does NOT run
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
# reads that leading double-dollar as its own PID, so a bound would compare a
# string like 1234{i} against 60. The bare form is the only spelling that is
# correct on BOTH paths.
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
# name") that fails this whole Kustomization — and this Kustomization gates every
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
# comment. That is why this note names the three forms in words rather than
# showing them: an earlier draft of it spelled them out literally and broke
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
# clusters/base/apps-vms.yaml describes.)
command:
- /bin/sh
- -c
- |
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
i=0
until [ -f /certs/client/ca.pem ]; do
i=$((i + 1))
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
echo "waiting for dind TLS certs ($i)"; sleep 1
done
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
j=0
until nc -z -w1 127.0.0.1 2376; do
j=$((j + 1))
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
done
exec /sbin/tini -- /opt/act/run.sh
env:
- name: GITEA_INSTANCE_URL
value: https://gitea.${BASE_DOMAIN}
@@ -199,6 +198,52 @@ spec:
memory: 128Mi
limits:
memory: 512Mi
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
#
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
# ordering that the loops above hand-roll — which also removes the envsubst hazard
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
# Shape to build:
# 1. Move this whole container into `initContainers:` as the FIRST entry.
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
# initContainer a sidecar: it starts before the regular containers, keeps
# running alongside them, and is terminated after them.
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
# it would re-open the same "port accepting but TLS/certs not ready" gap that
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
# preserves today's 60s cap.
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
# `runner` can drop its `command:` override completely and run the image's
# default entrypoint.
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
# entire repository. If the conversion is wrong, nothing can merge — so it must land
# alone, where the failure is unambiguous and not entangled with anything else, and
# only after a real workflow run has been observed green on the converted pod.
- name: dind
image: docker:27-dind
securityContext:
privileged: true
env:
# Generate + publish TLS certs into the shared emptyDir; dockerd then listens
# on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
- name: DOCKER_TLS_CERTDIR
value: /certs
volumeMounts:
- name: certs
mountPath: /certs
- name: dind-storage
mountPath: /var/lib/docker
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 2Gi
volumes:
- name: data
persistentVolumeClaim:
-208
View File
@@ -1,208 +0,0 @@
---
# Namespace (trust zone; default-deny + policies apply)
apiVersion: v1
kind: Namespace
metadata:
name: keycloak
labels:
homelab.keithjackson.dev/app-namespace: 'true'
gateway-access: shared
annotations:
homelab.keithjackson.dev/backup-policy: daily-offcluster
---
# Default-deny baseline (R-NET-01): deny all, allow only DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: keycloak
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# PVC keycloak-db (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: keycloak-db
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 10Gi
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: keycloak
template:
metadata:
labels:
app.kubernetes.io/name: keycloak
homelab.keithjackson.dev/egress-class: internet
homelab.keithjackson.dev/auth-mode: native
spec:
containers:
- name: keycloak
image: quay.io/keycloak/keycloak
ports:
- containerPort: 8080
volumeMounts:
- name: db
mountPath: /data/db
volumes:
- name: db
persistentVolumeClaim:
claimName: keycloak-db
---
# Service
apiVersion: v1
kind: Service
metadata:
name: keycloak
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
selector:
app.kubernetes.io/name: keycloak
ports:
- name: port-8080
port: 8080
targetPort: 8080
---
# CiliumNetworkPolicy egress allow (R-NET-02): keycloak-allow-keycloak-db-5432
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: keycloak-allow-keycloak-db-5432
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: keycloak
egress:
- toEndpoints:
- matchLabels:
app.kubernetes.io/name: keycloak-db
k8s:io.kubernetes.pod.namespace: keycloak-db
toPorts:
- ports:
- port: '5432'
protocol: TCP
---
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): keycloak-allow-gateway-ingress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: keycloak-allow-gateway-ingress
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: keycloak
ingress:
- fromEntities:
- ingress
toPorts:
- ports:
- port: '8080'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: keycloak-allow-internet-egress
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: keycloak
egress:
- toEntities:
- world
---
# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: keycloak-8080
namespace: keycloak
labels:
app.kubernetes.io/name: keycloak
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- keycloak.${BASE_DOMAIN}
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: keycloak
port: 8080
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: keycloak-slis
namespace: keycloak
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: keycloak
spec:
groups:
- name: keycloak.slis
rules:
- alert: keycloak-down
expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
for: 5m
labels:
severity: critical
service: keycloak
annotations:
summary: keycloak is down
description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
+22 -54
View File
@@ -35,34 +35,6 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume nextcloud-legacy-archive - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: nextcloud-legacy-archive
labels:
app.kubernetes.io/name: nextcloud
spec:
capacity:
storage: 270Gi
accessModes:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/nextcloud
readOnly: true
claimRef:
namespace: nextcloud
name: nextcloud-legacy-archive
---
# PVC nextcloud-data (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -79,23 +51,6 @@ spec:
requests:
storage: 20Gi
---
# PVC nextcloud-legacy-archive (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nextcloud-legacy-archive
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
accessModes:
- ReadOnlyMany
storageClassName: ''
volumeName: nextcloud-legacy-archive
resources:
requests:
storage: 270Gi
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -164,10 +119,6 @@ spec:
volumeMounts:
- name: data
mountPath: /var/www/html
- name: legacy-archive
mountPath: /mnt/legacy-nextcloud
subPath: nextcloud
readOnly: true
lifecycle:
postStart:
exec:
@@ -181,9 +132,6 @@ spec:
- name: data
persistentVolumeClaim:
claimName: nextcloud-data
- name: legacy-archive
persistentVolumeClaim:
claimName: nextcloud-legacy-archive
---
# Service
apiVersion: v1
@@ -290,6 +238,26 @@ spec:
- name: nextcloud
port: 80
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nextcloud
namespace: nextcloud
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: nextcloud
spec:
selector:
matchLabels:
app.kubernetes.io/name: nextcloud
namespaceSelector:
matchNames:
- nextcloud
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -304,11 +272,11 @@ spec:
- name: nextcloud.slis
rules:
- alert: nextcloud-down
expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
expr: up{job="nextcloud"} == 0
for: 5m
labels:
severity: critical
service: nextcloud
annotations:
summary: nextcloud is down
description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'nextcloud has been unreachable for 5m (SLI: up).'
+25 -65
View File
@@ -35,33 +35,6 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume radarr-media - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: radarr-media
labels:
app.kubernetes.io/name: radarr
spec:
capacity:
storage: 9Ti
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
claimRef:
namespace: radarr
name: radarr-media
---
# PVC radarr-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -78,23 +51,6 @@ spec:
requests:
storage: 5Gi
---
# PVC radarr-media (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: radarr-media
namespace: radarr
labels:
app.kubernetes.io/name: radarr
spec:
accessModes:
- ReadWriteMany
storageClassName: ''
volumeName: radarr-media
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -118,25 +74,16 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
homelab.keithjackson.dev/auth-groups: media-users
spec:
containers:
- name: radarr
image: lscr.io/linuxserver/radarr:5.21.1
ports:
- containerPort: 7878
env:
- name: PUID
value: '1000'
- name: PGID
value: '3000'
- name: TZ
value: Etc/UTC
volumeMounts:
- name: config
mountPath: /config
- name: media
mountPath: /media
mountPath: /data/config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -150,7 +97,6 @@ spec:
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
@@ -202,15 +148,10 @@ spec:
port: http
initialDelaySeconds: 5
periodSeconds: 15
securityContext:
fsGroup: 3000
volumes:
- name: config
persistentVolumeClaim:
claimName: radarr-config
- name: media
persistentVolumeClaim:
claimName: radarr-media
---
# Service
apiVersion: v1
@@ -347,8 +288,7 @@ metadata:
app.kubernetes.io/name: radarr
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
external-dns.alpha.kubernetes.io/controller: none
homelab.keithjackson.dev/auth-groups: media-users
spec:
parentRefs:
- name: shared
@@ -365,6 +305,26 @@ spec:
- name: radarr
port: 4180
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: radarr
namespace: radarr
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: radarr
spec:
selector:
matchLabels:
app.kubernetes.io/name: radarr
namespaceSelector:
matchNames:
- radarr
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -379,11 +339,11 @@ spec:
- name: radarr.slis
rules:
- alert: radarr-down
expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
expr: up{job="radarr"} == 0
for: 5m
labels:
severity: critical
service: radarr
annotations:
summary: radarr is down
description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'radarr has been unreachable for 5m (SLI: up).'
@@ -5,15 +5,6 @@
# credential from its own namespace (a Pod cannot read a Secret in another namespace).
# access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
# credentials; MUST stay stable across restarts or stored secrets become undecryptable.
# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
# `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
# injects that copy into the client), same cross-namespace mirror pattern as db-password above.
# It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
# SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
# URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
# so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
# catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
# back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
# Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
apiVersion: v1
kind: Secret
@@ -24,11 +15,11 @@ metadata:
app.kubernetes.io/name: semaphore
type: Opaque
stringData:
admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
sops:
kms: []
gcp_kms: []
@@ -38,14 +29,14 @@ sops:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-07T07:01:50Z"
mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
lastmodified: "2026-08-27T22:12:32Z"
mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+24 -7
View File
@@ -118,13 +118,11 @@ spec:
key: access-key-encryption
- name: SEMAPHORE_PORT
value: :3000
- name: SEMAPHORE_OIDC_CLIENT_SECRET
- name: SEMAPHORE_OIDC_PROVIDERS
valueFrom:
secretKeyRef:
name: semaphore-bootstrap
key: oidc-client-secret
- name: SEMAPHORE_OIDC_PROVIDERS
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
key: oidc-providers-json
volumeMounts:
- name: config
mountPath: /etc/semaphore
@@ -227,7 +225,6 @@ metadata:
annotations:
homelab.keithjackson.dev/auth-mode: oidc
homelab.keithjackson.dev/auth-groups: platform-admins
external-dns.alpha.kubernetes.io/controller: none
spec:
parentRefs:
- name: shared
@@ -244,6 +241,26 @@ spec:
- name: semaphore
port: 3000
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: semaphore
namespace: semaphore
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: semaphore
spec:
selector:
matchLabels:
app.kubernetes.io/name: semaphore
namespaceSelector:
matchNames:
- semaphore
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -258,11 +275,11 @@ spec:
- name: semaphore.slis
rules:
- alert: semaphore-down
expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
expr: up{job="semaphore"} == 0
for: 5m
labels:
severity: critical
service: semaphore
annotations:
summary: semaphore is down
description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'semaphore has been unreachable for 5m (SLI: up).'
+25 -65
View File
@@ -35,33 +35,6 @@ spec:
- protocol: TCP
port: 53
---
# PersistentVolume sonarr-media - static external NFS (bulk tier)
apiVersion: v1
kind: PersistentVolume
metadata:
name: sonarr-media
labels:
app.kubernetes.io/name: sonarr
spec:
capacity:
storage: 9Ti
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ''
mountOptions:
- nfsvers=4.1
- hard
- noatime
- timeo=600
- retrans=2
nfs:
server: ${BULK_NFS_SERVER}
path: /mnt/Storage/plex
claimRef:
namespace: sonarr
name: sonarr-media
---
# PVC sonarr-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
@@ -78,23 +51,6 @@ spec:
requests:
storage: 5Gi
---
# PVC sonarr-media (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sonarr-media
namespace: sonarr
labels:
app.kubernetes.io/name: sonarr
spec:
accessModes:
- ReadWriteMany
storageClassName: ''
volumeName: sonarr-media
resources:
requests:
storage: 9Ti
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
@@ -118,25 +74,16 @@ spec:
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
homelab.keithjackson.dev/auth-groups: media-users
spec:
containers:
- name: sonarr
image: lscr.io/linuxserver/sonarr:4.0.15
ports:
- containerPort: 8989
env:
- name: PUID
value: '1000'
- name: PGID
value: '3000'
- name: TZ
value: Etc/UTC
volumeMounts:
- name: config
mountPath: /config
- name: media
mountPath: /media
mountPath: /data/config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
@@ -150,7 +97,6 @@ spec:
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
@@ -202,15 +148,10 @@ spec:
port: http
initialDelaySeconds: 5
periodSeconds: 15
securityContext:
fsGroup: 3000
volumes:
- name: config
persistentVolumeClaim:
claimName: sonarr-config
- name: media
persistentVolumeClaim:
claimName: sonarr-media
---
# Service
apiVersion: v1
@@ -347,8 +288,7 @@ metadata:
app.kubernetes.io/name: sonarr
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
external-dns.alpha.kubernetes.io/controller: none
homelab.keithjackson.dev/auth-groups: media-users
spec:
parentRefs:
- name: shared
@@ -365,6 +305,26 @@ spec:
- name: sonarr
port: 4180
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: sonarr
namespace: sonarr
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: sonarr
spec:
selector:
matchLabels:
app.kubernetes.io/name: sonarr
namespaceSelector:
matchNames:
- sonarr
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -379,11 +339,11 @@ spec:
- name: sonarr.slis
rules:
- alert: sonarr-down
expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
expr: up{job="sonarr"} == 0
for: 5m
labels:
severity: critical
service: sonarr
annotations:
summary: sonarr is down
description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'sonarr has been unreachable for 5m (SLI: up).'
-245
View File
@@ -1,245 +0,0 @@
# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
# the flattened manifests are the RENDERED ones Flux applies.
#
# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
#
# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
# replica, and FIRING when the signal is ABSENT.
#
# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
# Deployment this ServiceDefinition would otherwise imply — so its rule reads
# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
# = 2, and the only Deployment in that namespace is keycloak-operator.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
# ===========================================================================================
# 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
# ===========================================================================================
- interval: 1m
name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
input_series:
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
values: '2+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
values: '1+0x20'
# The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
# the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
# namespace only (not the object) would be satisfied — or triggered — by these instead of by
# the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
- series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
values: '1+0x20'
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
values: '3+0x20'
alert_rule_test:
- eval_time: 10m
alertname: keycloak-down
exp_alerts: []
- eval_time: 10m
alertname: nextcloud-down
exp_alerts: []
- eval_time: 10m
alertname: radarr-down
exp_alerts: []
- eval_time: 10m
alertname: semaphore-down
exp_alerts: []
- eval_time: 10m
alertname: sonarr-down
exp_alerts: []
# ===========================================================================================
# 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
# This is the anti-masking assertion: the neighbours in that namespace stay ready.
# ===========================================================================================
- interval: 1m
name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
input_series:
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
values: '1+0x20'
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
values: '3+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: keycloak-down
exp_alerts:
- exp_labels:
severity: critical
service: keycloak
namespace: keycloak
statefulset: keycloak
exp_annotations:
summary: "keycloak is down"
description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
- eval_time: 10m
alertname: nextcloud-down
exp_alerts: []
- eval_time: 10m
alertname: radarr-down
exp_alerts: []
- eval_time: 10m
alertname: semaphore-down
exp_alerts: []
- eval_time: 10m
alertname: sonarr-down
exp_alerts: []
# ===========================================================================================
# 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
# ===========================================================================================
- interval: 1m
name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
input_series:
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
values: '2+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
values: '0+0x20'
alert_rule_test:
- eval_time: 10m
alertname: nextcloud-down
exp_alerts:
- exp_labels:
severity: critical
service: nextcloud
namespace: nextcloud
deployment: nextcloud
exp_annotations:
summary: "nextcloud is down"
description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: radarr-down
exp_alerts:
- exp_labels:
severity: critical
service: radarr
namespace: radarr
deployment: radarr
exp_annotations:
summary: "radarr is down"
description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: semaphore-down
exp_alerts:
- exp_labels:
severity: critical
service: semaphore
namespace: semaphore
deployment: semaphore
exp_annotations:
summary: "semaphore is down"
description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: sonarr-down
exp_alerts:
- exp_labels:
severity: critical
service: sonarr
namespace: sonarr
deployment: sonarr
exp_annotations:
summary: "sonarr is down"
description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: keycloak-down
exp_alerts: []
# ===========================================================================================
# 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
# only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
# which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
# ===========================================================================================
- interval: 1m
name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
input_series:
# One unrelated series so the test has an input at all; no wave-1 liveness series exists.
- series: 'up{job="kube-state-metrics"}'
values: '0+0x20'
alert_rule_test:
- eval_time: 10m
alertname: keycloak-down
exp_alerts:
- exp_labels:
severity: critical
service: keycloak
namespace: keycloak
statefulset: keycloak
exp_annotations:
summary: "keycloak is down"
description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
- eval_time: 10m
alertname: nextcloud-down
exp_alerts:
- exp_labels:
severity: critical
service: nextcloud
namespace: nextcloud
deployment: nextcloud
exp_annotations:
summary: "nextcloud is down"
description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: radarr-down
exp_alerts:
- exp_labels:
severity: critical
service: radarr
namespace: radarr
deployment: radarr
exp_annotations:
summary: "radarr is down"
description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: semaphore-down
exp_alerts:
- exp_labels:
severity: critical
service: semaphore
namespace: semaphore
deployment: semaphore
exp_annotations:
summary: "semaphore is down"
description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time: 10m
alertname: sonarr-down
exp_alerts:
- exp_labels:
severity: critical
service: sonarr
namespace: sonarr
deployment: sonarr
exp_annotations:
summary: "sonarr is down"
description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+22 -2
View File
@@ -233,6 +233,26 @@ spec:
- name: seerr
port: 5055
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: seerr
namespace: seerr
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: seerr
spec:
selector:
matchLabels:
app.kubernetes.io/name: seerr
namespaceSelector:
matchNames:
- seerr
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
@@ -247,11 +267,11 @@ spec:
- name: seerr.slis
rules:
- alert: seerr-down
expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
expr: up{job="seerr"} == 0
for: 5m
labels:
severity: critical
service: seerr
annotations:
summary: seerr is down
description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
description: 'seerr has been unreachable for 5m (SLI: up).'
-69
View File
@@ -1,69 +0,0 @@
# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
# manifest is the RENDERED one Flux applies.
#
# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
#
# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
# signal-vanishes case.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
- interval: 1m
name: wave-2 healthy - seerr has an available replica - quiet
input_series:
- series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: seerr-down
exp_alerts: []
- interval: 1m
name: wave-2 seerr has no available replica - seerr-down fires
input_series:
- series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
values: '0+0x20'
alert_rule_test:
- eval_time: 10m
alertname: seerr-down
exp_alerts:
- exp_labels:
severity: critical
service: seerr
namespace: seerr
deployment: seerr
exp_annotations:
summary: "seerr is down"
description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
# Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
# already be selecting — the distinction an inert rule cannot make.
promql_expr_test:
- expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
eval_time: 2m
exp_samples:
- labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
value: 0
- interval: 1m
name: wave-2 seerr liveness series ABSENT - seerr-down still fires
input_series:
- series: 'up{job="kube-state-metrics"}'
values: '1+0x20'
alert_rule_test:
- eval_time: 10m
alertname: seerr-down
exp_alerts:
- exp_labels:
severity: critical
service: seerr
namespace: seerr
deployment: seerr
exp_annotations:
summary: "seerr is down"
description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+50 -4
View File
@@ -11,7 +11,8 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
## Files
| File | Purpose |
|---|---|
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. |
| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. |
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
@@ -74,6 +75,49 @@ flux get kustomizations -A # all 11 must return
**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must
stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with
`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB
`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the
HelmRelease and blocks `platform` plus every tier that `dependsOn` it.
The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's
Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only
because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git:
```
helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
| yq 'select(.kind == "CustomResourceDefinition")'
| kubectl apply --server-side --force-conflicts
```
- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are
read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver
**constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax
helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump
of that one line carries this step with it. There is nothing here that can drift.
- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the
HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the
HelmRelease over field managers. The `yq` filter is load-bearing.
- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's
hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints
`serverside-applied` and exits 0.
- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not
`null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and
exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is
waited on for `condition=Established`. There is **no skip flag** on purpose.
- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads
repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step.
Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs):
```bash
# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
bash bootstrap/flux/preapply-crds.sh
```
## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
@@ -98,7 +142,7 @@ tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1
▼ (final Ansible play calls the entrypoints below)
1. bootstrap/sops/age-keygen.sh # operator generates age key OUTSIDE repo
2. bootstrap/sops/age-key-secret.md # operator installs private key as sops-age Secret
3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev
3. bootstrap/flux/bootstrap.sh # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
│
▼
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
@@ -118,8 +162,10 @@ live in `infra/devcontainer/versions.env`).
template). `bootstrap.sh` **fails fast** if this Secret is absent.
3. `bootstrap/flux/bootstrap.sh` (this entrypoint).
`bootstrap.sh` is idempotent-friendly (`flux install` and `kubectl apply -k` re-run safely) and
never handles key material — it only *verifies* the Secret exists.
`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and
`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret
exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render;
both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo.
## Handoff — clusters owner / PM (files this worker may not edit)
+31 -8
View File
@@ -33,10 +33,19 @@
# the single apps.yaml was retired). Letting
# bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
# applying the authored clusters/dev overlay is the reconciling path.
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
# 2. CRD pre-apply — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`
# (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit,
# which STALLS the HelmRelease and blocks the `platform` Kustomization and
# every tier below it). With CRD install off, NOTHING else installs them, so
# a cold cluster can never reconcile `platform` without this step. The chart
# version is DERIVED from that same HelmRelease — no second pin. Dev only
# ever had these CRDs because they were applied BY HAND; this is that step,
# in git, idempotent and version-derived. See ./preapply-crds.sh.
# 3. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
# decryption works on first pass (../sops/age-key-secret.md). This script
# only VERIFIES it exists; it never handles key material.
# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
# 4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -71,7 +80,10 @@ repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
for bin in flux kubectl; do
# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored
# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the
# Flux controllers are installed, instead of half-way through.
for bin in flux kubectl helm yq; do
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
done
@@ -99,7 +111,7 @@ done
#
# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
flux_install_args=(--namespace="${FLUX_NAMESPACE}")
[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
@@ -130,18 +142,29 @@ else
kubectl apply -f "${render_dir}/rendered.yaml"
fi
# --- 2. verify the out-of-band age key Secret exists -------------------------
# --- 2. pre-apply the CRDs Helm is not allowed to install ---------------------
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs
# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's
# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall.
# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies
# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD
# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a
# cluster that looks bootstrapped and reconciles nothing.
echo "==> [2/4] pre-applying chart CRDs (server-side)"
bash "${script_dir}/preapply-crds.sh"
# --- 3. verify the out-of-band age key Secret exists -------------------------
# We DO NOT create it here (it holds the private key). The operator installs it per
# ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op.
echo "==> [2/3] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then
echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2
echo " Install the age private key first: bootstrap/sops/age-key-secret.md" >&2
exit 1
fi
# --- 3. apply the authored environment entrypoint ----------------------------
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
# --- 4. apply the authored environment entrypoint ----------------------------
echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}"
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
echo
+169
View File
@@ -0,0 +1,169 @@
#!/usr/bin/env bash
#
# preapply-crds.sh — server-side pre-apply of the CRDs that Helm is NOT allowed to install.
# ENVIRONMENT-AGNOSTIC: it reads the chart coordinates out of the authored HelmRelease and talks to
# whatever cluster $KUBECONFIG points at, so dev and prod run the SAME step with no fork (R-ENV-02).
# Called by ./bootstrap.sh (step 2/4); also safe to run STANDALONE against an already-bootstrapped
# cluster (that is how prod gets unblocked without re-running the whole bootstrap).
#
# R-GITOPS-01 (the graph must reconcile from git, not from someone's shell history) · R-ENV-02.
# =============================================================================
# *** AUTHORED, NOT RUN. Agents must not execute it — it MUTATES a cluster. ***
# =============================================================================
#
# WHY THIS EXISTS
#
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`. That is deliberate:
# the Kyverno CRD bundle is ~5.6 MiB, far past kubectl's 256 KiB last-applied-configuration
# annotation limit, so Helm's client-side apply fails with "metadata.annotations: Too long" and
# STALLS the HelmRelease — which blocks the `platform` Kustomization and every tier that dependsOn
# it (the kps-cascade failure mode; see the HelmRelease's own comment).
#
# The consequence nobody automated: with CRD install off, NOTHING installs those CRDs. Kyverno's
# Deployments come up but its API types do not exist, so `platform` never goes Ready on a cold
# cluster. The dev cluster only has them because they were once applied BY HAND — exactly the
# undocumented manual step this repo exists to eliminate. This script is that step, in git,
# version-derived and idempotent.
#
# HOW THE VERSION IS DERIVED (there is deliberately NO second copy of it)
#
# Everything comes out of the HelmRelease/HelmRepository pair that Flux itself reconciles:
#
# chart name spec.chart.spec.chart (HelmRelease)
# chart version spec.chart.spec.version (HelmRelease) e.g. "3.9.x"
# chart repo url spec.url (HelmRepository)
# namespace metadata.namespace (HelmRelease)
#
# The version is a semver CONSTRAINT, not a fixed tag. `helm template --version` accepts the same
# constraint syntax Flux's helm-controller resolves (both use Masterminds/semver), so passing it
# through UNCHANGED renders the very chart version Flux will install. Renovate bumps one line in
# one file and this step follows automatically — it cannot drift, because there is nothing here to
# drift from.
#
# WHY ONLY THE CRD DOCUMENTS ARE APPLIED
#
# The by-hand command recorded in the HelmRelease comment piped the WHOLE `helm template` output
# into kubectl. Do not do that: it would create Deployments/Services/webhooks outside Helm's
# ownership, which then fight the HelmRelease over field managers and leave orphans behind. We
# filter to `kind: CustomResourceDefinition` and apply nothing else. Values are the chart DEFAULTS
# (plus an explicit crds.install=true) — NOT the repo's values, which switch CRD rendering off.
#
# IDEMPOTENCY
#
# `kubectl apply --server-side --force-conflicts` is a declarative upsert: the first run creates,
# every later run reports "serverside-applied" and exits 0, and field ownership is taken from
# whatever previously wrote those objects (that is what --force-conflicts is for — it resolves the
# hand-applied dev CRDs' `kubectl-client-side-apply` manager without a human deciding anything).
# Server-side apply does not write the last-applied annotation, which is the whole reason the
# 5.6 MiB bundle can be applied at all.
#
# LOUD FAILURE (this script must never degrade to a successful no-op)
#
# `set -euo pipefail`, plus three explicit guards, because a silent skip here reproduces the exact
# class of bug this repo has been bitten by more than once:
# 1. every derived coordinate must be non-empty and not "null" — a moved YAML path fails here;
# 2. the rendered CRD count must be > 0 — if a future chart moves/renames its CRD templates the
# filter would match nothing and `kubectl apply` on an empty stream EXITS 0. Guarded;
# 3. after applying, every CRD is waited on for condition=Established — so "applied" is proven
# to mean "the API type exists", not merely "kubectl returned 0".
#
set -euo pipefail
# --- config (env-overridable) ------------------------------------------------
# The single source of truth for the chart coordinates. Repo-relative on purpose: this is the file
# Flux reconciles, so it is the file this step must agree with.
CRD_HELMRELEASE="${CRD_HELMRELEASE:-platform/kyverno/admission/helmrelease.yaml}"
# How long to wait for the applied CRDs to become Established (apiextensions publishes the type).
CRD_ESTABLISH_TIMEOUT="${CRD_ESTABLISH_TIMEOUT:-120s}"
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
echo "==> CRD pre-apply — source of truth: ${CRD_HELMRELEASE}"
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
# helm renders the bundle, yq selects the CRD documents, kubectl applies them server-side. All three
# are baked into infra/devcontainer (HELM_VERSION / YQ_VERSION / KUBECTL_VERSION in versions.env).
for bin in helm yq kubectl; do
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
done
helmrelease="${repo_root}/${CRD_HELMRELEASE}"
[ -f "${helmrelease}" ] || {
echo "ERROR: ${helmrelease} not found — CRD_HELMRELEASE points nowhere." >&2
exit 1
}
# --- 1. derive the chart coordinates from the authored manifests --------------
chart_name="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.chart' "${helmrelease}")"
chart_version="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "${helmrelease}")"
chart_ns="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .metadata.namespace' "${helmrelease}")"
chart_repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' "${helmrelease}")"
# Guard 1: a moved/renamed YAML path must STOP the bootstrap, not render a bogus chart. yq prints
# "null" for a missing path and an empty string for a missing document, so both are rejected.
for pair in "chart:${chart_name}" "version:${chart_version}" "namespace:${chart_ns}" "repo:${chart_repo}"; do
key="${pair%%:*}"; val="${pair#*:}"
case "${val}" in
""|null|*$'\n'*)
echo "ERROR: could not derive a single '${key}' from ${CRD_HELMRELEASE} (got: '${val}')." >&2
echo " The HelmRelease/HelmRepository shape changed — fix the yq paths in $(basename "${BASH_SOURCE[0]}")." >&2
exit 1
;;
esac
done
echo " chart=${chart_name} version=${chart_version} repo=${chart_repo} namespace=${chart_ns}"
echo " (version is the constraint Flux resolves — NOT a second copy pinned here)"
# --- 2. render the chart and keep ONLY the CRD documents ----------------------
render_dir="$(mktemp -d)"
trap 'rm -rf "${render_dir}"' EXIT
crd_file="${render_dir}/crds.yaml"
# --include-crds covers charts that ship a native crds/ dir; crds.install=true covers Kyverno, whose
# CRDs are subchart TEMPLATES that the repo's own values switch off. Chart defaults otherwise.
echo "==> rendering ${chart_name} ${chart_version} (CRD documents only)"
helm template "${chart_name}" "${chart_name}" \
--repo "${chart_repo}" \
--version "${chart_version}" \
--namespace "${chart_ns}" \
--include-crds \
--set crds.install=true \
| yq eval-all 'select(.kind == "CustomResourceDefinition")' - > "${crd_file}"
# Blank/"null" lines are dropped on purpose: an EMPTY render makes `yq .metadata.name` emit one
# empty line, which would otherwise count as a CRD and sail straight past guard 2 below.
mapfile -t crd_names < <(yq eval-all --no-doc '.metadata.name' "${crd_file}" | grep -vx -e '' -e 'null' || true)
crd_count="${#crd_names[@]}"
# Guard 2: an empty stream applies cleanly and exits 0 — the silent-no-op trap. Refuse it.
[ "${crd_count}" -gt 0 ] || {
echo "ERROR: rendered 0 CustomResourceDefinitions from ${chart_name} ${chart_version}." >&2
echo " The chart's CRD layout or the crds.install value key changed. Refusing to continue:" >&2
echo " applying nothing would look like success and leave 'platform' blocked forever." >&2
exit 1
}
echo " ${crd_count} CRDs, $(wc -c < "${crd_file}") bytes (>256 KiB — this is why it must be server-side)"
# --- 3. server-side apply (idempotent upsert) ---------------------------------
# --server-side: no last-applied annotation, so the size limit does not apply.
# --force-conflicts: adopt fields owned by an earlier manager (e.g. dev's hand-applied CRDs, or
# Helm's own manager if a past chart version did install them). Makes a re-run a plain no-op.
echo "==> kubectl apply --server-side --force-conflicts (${crd_count} CRDs)"
kubectl apply --server-side --force-conflicts -f "${crd_file}"
# --- 4. prove the API types actually exist -----------------------------------
# Without this, "kubectl returned 0" is the only evidence. Establishment is what Kyverno's
# controllers and the platform Kustomization actually need.
echo "==> waiting for condition=Established (timeout ${CRD_ESTABLISH_TIMEOUT})"
crd_refs=()
for n in "${crd_names[@]}"; do crd_refs+=("crd/${n}"); done
kubectl wait --for=condition=Established --timeout="${CRD_ESTABLISH_TIMEOUT}" \
"${crd_refs[@]}" >/dev/null || {
echo "ERROR: not all ${crd_count} CRDs reached Established within ${CRD_ESTABLISH_TIMEOUT}." >&2
echo " Inspect: kubectl get crd | grep ${chart_name}" >&2
exit 1
}
echo "==> ${crd_count} ${chart_name} CRDs applied and Established."
+988
View File
@@ -0,0 +1,988 @@
#!/usr/bin/env bash
#
# standup.sh — bring a bootstrapped cluster's reconciliation graph all the way up.
#
# ENVIRONMENT-AGNOSTIC (R-ENV-02): it reads everything it needs from the cluster it is pointed at and
# from the authored manifests. There is no dev/prod fork. What makes it safe to run against prod is
# the MANDATORY --expect-server guard, not a name in the filename.
#
# WHY THIS EXISTS
# `bootstrap/flux/bootstrap.sh` installs Flux and applies the graph. It does NOT finish the job when
# tiers have been deliberately suspended (a DNS/TLS migration window, an incident recovery) or when
# a HelmRelease has wedged. Those last steps were living in shell history and in a human's head,
# which is the same failure the Kyverno CRD pre-apply had. This is that sequence, in git, ordered,
# idempotent, and gated.
#
# THE ORDERING IS LOAD-BEARING. Read infra/PROD-STANDUP.md before deviating from it.
#
# USAGE
# bootstrap/standup.sh --kubeconfig <path> --expect-server <url> [--confirm-destructive] <step...>
#
# steps: preflight crds unwedge externaldns s3check resume verify (or: all)
#
# --expect-server is REQUIRED and is compared against the live API server before ANY command runs.
# Pointing this at the wrong cluster is the one mistake with unbounded blast radius: dev is live and
# shares a Cloudflare zone with prod. The guard is not optional and there is no override flag.
#
# --confirm-destructive is required by `unwedge` only. Everything else is additive or reversible.
#
# --s3-endpoint <url> makes `s3check` probe a CANDIDATE backup endpoint instead of the live
# cluster-vars value, so a proposed target can be PROVEN before it is authored into cluster-vars.
# `s3check` writes and then deletes ONE throwaway object under `<prefix>/.s3check/`; it changes
# nothing else and applies nothing. See infra/BACKUP-R2.md.
#
set -euo pipefail
# ---------------------------------------------------------------------------- args
KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 S3_ENDPOINT_OVERRIDE=""
STEPS=()
while [ $# -gt 0 ]; do
case "$1" in
--kubeconfig) KUBECONFIG_ARG="${2:?--kubeconfig needs a value}"; shift 2 ;;
--expect-server) EXPECT_SERVER="${2:?--expect-server needs a value}"; shift 2 ;;
--confirm-destructive) CONFIRM_DESTRUCTIVE=1; shift ;;
--s3-endpoint) S3_ENDPOINT_OVERRIDE="${2:?--s3-endpoint needs a value}"; shift 2 ;;
-h|--help) sed -n '2,30p' "$0"; exit 0 ;;
-*) echo "ERROR: unknown flag '$1'" >&2; exit 2 ;;
*) STEPS+=("$1"); shift ;;
esac
done
[ -n "${KUBECONFIG_ARG}" ] || { echo "ERROR: --kubeconfig is required." >&2; exit 2; }
[ -n "${EXPECT_SERVER}" ] || { echo "ERROR: --expect-server is required (safety guard)." >&2; exit 2; }
[ "${#STEPS[@]}" -gt 0 ] || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns s3check resume verify all" >&2; exit 2; }
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/.." && pwd))"
cd "${repo_root}"
export KUBECONFIG="${KUBECONFIG_ARG}"
[ -f "${KUBECONFIG}" ] || { echo "ERROR: kubeconfig not found: ${KUBECONFIG}" >&2; exit 1; }
# Where DNS snapshots go. This MUST be a path that survives between invocations: each `dev.sh run` is
# a FRESH container, so a $TMPDIR default silently threw the baseline away and made `verify`'s
# before/after comparison — the check that detects damage to the OTHER cluster's records — impossible
# to run across steps. The repo is bind-mounted, so a repo-relative dir persists on the host.
# Gitignored: these are run artifacts, not authored state.
SNAP_DIR="${STANDUP_SNAP_DIR:-${repo_root}/.standup-dns}"
mkdir -p "${SNAP_DIR}"
say() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
warn() { printf '\033[33mWARN: %s\033[0m\n' "$*" >&2; }
for bin in kubectl flux helm yq sops; do
command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer."
done
# ------------------------------------------------------------------ THE SAFETY GUARD
# Checked once, before any step, and re-checked by every mutating helper. The failure mode this
# prevents is running a prod bring-up against the LIVE dev cluster, whose public DNS records prod
# has already destroyed once (2026-09-04).
live_server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')"
[ "${live_server}" = "${EXPECT_SERVER}" ] || die \
"TARGET MISMATCH — refusing to run.
--expect-server : ${EXPECT_SERVER}
live API server : ${live_server}
The kubeconfig does not point where you think it does. Nothing has been changed."
say "target confirmed: ${live_server}"
# Dependency order. Flux enforces the edges itself via dependsOn; resuming in this order simply means
# each wait is meaningful instead of blocking on a dependency that is still suspended.
# flux-system -> cilium -> platform -> platform-config -> platform-config-cdi
# -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2 -> apps-vms
#
# `platform-config-cdi` MUST stay in this list and MUST stay before apps-vms. It carries the CDI
# StorageProfiles, which were split out of platform-config because StorageProfile's CRD does not exist
# until cdi-operator has reconciled the CDI CR (clusters/base/platform-config-cdi.yaml). apps-vms
# dependsOn it: apps/vms/unifi provisions on `longhorn-vm`, and without the profile applied CDI infers
# ReadWriteOnce instead of the authored ReadWriteMany — permanently non-live-migratable for that disk.
# Omitting it here would make `resume` return success without ever waiting on the tier that guarantees it.
TIER_ORDER=(platform platform-config platform-config-cdi oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which is
# an intentionally-invalid placeholder until the operator supplies the real R2 endpoint.
cvar() { kubectl -n flux-system get cm cluster-vars -o jsonpath="{.data.$1}" 2>/dev/null || true; }
k_ready() { kubectl -n flux-system get kustomization "$1" \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true; }
k_msg() { kubectl -n flux-system get kustomization "$1" \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' 2>/dev/null || true; }
k_susp() { kubectl -n flux-system get kustomization "$1" -o jsonpath='{.spec.suspend}' 2>/dev/null || true; }
wait_ready() { # <kustomization> <timeout-seconds>
local n="$1" t="${2:-600}" i=0
while [ "$i" -lt "$t" ]; do
[ "$(k_ready "$n")" = "True" ] && { info "$n Ready (${i}s)"; return 0; }
sleep 10; i=$((i+10))
done
warn "$n did NOT reach Ready in ${t}s: $(k_msg "$n")"
return 1
}
# Snapshot the Cloudflare zone. This is how damage to the OTHER cluster's records becomes detectable.
# Queried against the Cloudflare API, never a local resolver: the workstation's own dnsmasq wildcard
# (address=/.home.keithjackson.dev/) answers the whole tree locally and produced a false
# healthy reading during the 2026-09-04 incident and hid the damage for a while.
dns_snapshot() { # <output-file>
local out="$1" tok zid page tp
tok="$(sops -d platform/externaldns/cloudflare-api-token.sops.yaml \
| yq '.stringData."api-token"' | tr -d '"')"
[ -n "${tok}" ] && [ "${tok}" != "null" ] || die "could not read the Cloudflare token (SOPS/age key available?)"
zid="$(curl -sS -H "Authorization: Bearer ${tok}" \
'https://api.cloudflare.com/client/v4/zones?name=keithjackson.dev' | yq '.result[0].id' | tr -d '"')"
[ -n "${zid}" ] && [ "${zid}" != "null" ] || die "Cloudflare zone lookup failed"
page=1; : > "${out}"
while :; do
local r; r="$(curl -sS -H "Authorization: Bearer ${tok}" \
"https://api.cloudflare.com/client/v4/zones/${zid}/dns_records?per_page=100&page=${page}")"
printf '%s' "${r}" | yq -o=json '.result' \
| yq -r '.[] | (.type + "|" + .name + "|" + .content)' | tr -d '"' >> "${out}"
tp="$(printf '%s' "${r}" | yq '.result_info.total_pages' | tr -d '"')"
[ "${page}" -ge "${tp:-1}" ] && break
page=$((page+1))
done
# Strip CRs, and sort under a FIXED collation. Both matter, and both were learned the hard way:
# - a stray \r makes every line differ from a baseline captured without one;
# - locale-dependent sort ordering puts `*.keithjackson.dev` in a different position on the host
# than in the container, so a diff reports pairs of IDENTICAL lines as changed. That produced a
# "RECORDS OUTSIDE ... CHANGED" alarm on a zone where nothing had changed at all.
# A regression check that cries wolf is worse than none — it trains you to skip the one alarm that
# matters. LC_ALL=C makes the ordering reproducible everywhere.
tr -d '\r' < "${out}" | grep -vE '^\s*$' | LC_ALL=C sort -o "${out}"
info "$(wc -l < "${out}") records -> ${out}"
}
dns_diff() { # <before> <after> — returns non-zero if anything changed
if diff -u "$1" "$2" > "${SNAP_DIR}/dns.diff"; then
info "Cloudflare zone IDENTICAL — nothing created, modified or deleted."
return 0
fi
warn "Cloudflare zone CHANGED:"; cat "${SNAP_DIR}/dns.diff"
return 1
}
# ======================================================================= s3check plumbing
# WHY THIS IS THIS BIG. The previous version of `s3check` curl'd /minio/health/live and reported
# REACHABLE. That probe could not pass against Cloudflare R2 (no such path), and reachability is not
# the property anyone cares about: a reachable endpoint with a wrong key, a missing bucket, or R2's
# checksum rejection still yields a velero tier that reports Ready while NO backup exists. What must
# be proven is the write path Velero and CNPG-barman actually use, with the credentials they actually
# hold, from the network position they actually occupy.
# Pinned. The AWS CLI is a deliberate choice over `minio/mc`:
# - it prints the S3 ERROR CODE (NoSuchBucket / AccessDenied / InvalidAccessKeyId / InvalidArgument),
# so a failure can be NAMED and turned into a fix instead of guessed at from an HTTP status;
# - it exposes the FLEXIBLE-CHECKSUM behaviour that is the R2 trap
# (AWS_REQUEST_CHECKSUM_CALCULATION), so the trap can be reproduced on purpose here instead of
# discovered by a silently-failing backup at 03:00;
# - its image carries bash + coreutils. `minio/mc` ships no sed/awk/diff — that minimalism has
# already broken one probe in this repo — and it cannot be made to send the checksum headers, so
# it cannot test for the trap at all.
S3CHECK_IMAGE="amazon/aws-cli:2.36.40"
S3CHECK_RUN_ID="$(date +%s)"
# Throwaway probe pods, tracked so they are removed on EVERY exit path (including `die`). A probe
# must never leave a pod behind on a live cluster.
S3CHECK_PODS=()
s3_cleanup_pods() {
local p
for p in ${S3CHECK_PODS[@]+"${S3CHECK_PODS[@]}"}; do
kubectl -n "${p%%/*}" delete pod "${p#*/}" --ignore-not-found --wait=false >/dev/null 2>&1 || true
done
S3CHECK_PODS=()
}
trap s3_cleanup_pods EXIT
# TRUE when the endpoint is not a usable target. `TBD-*` alone was WRONG once the placeholder became
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`: that does not START with TBD-, so the old
# prefix pattern matched nothing and `resume` would have resumed velero straight onto a placeholder
# hostname — the exact silent-no-backups failure the guard exists to prevent. Match anywhere.
s3_is_placeholder() { # <endpoint>
case "$1" in
""|*TBD*|*tbd-*|*REPLACE-ME*|*replace-me*|*CHANGE-ME*|*change-me*) return 0 ;;
esac
return 1
}
# Refuse any value that is not a plain URL/name fragment. This is BOTH a safety guard — these values
# are interpolated into the probe pod's JSON overrides AND into a shell script that runs inside it, so
# a quote, a backslash or a newline could rewrite either — and a correctness guard: a cluster-vars
# value with a trailing space or a stray quote is a defect, not something to paper over.
s3_safe_token() { # <label> <value>
case "$2" in
*[!A-Za-z0-9:/._-]*) die "refusing to probe: $1 contains a character outside [A-Za-z0-9:/._-].
That is either a corrupt cluster-vars value or an injection attempt. Value (quoted): $(printf '%q' "$2")" ;;
esac
}
# ---------------------------------------------------------------- the in-pod probe
# Emitted to the pod base64-encoded, so nothing in it has to survive JSON or shell quoting.
# CONTRACT WITH THE CALLER (this is what makes a silent no-op impossible):
# - every proven property prints exactly one `S3CHECK-OK[n] ...` line and increments n;
# - every failure prints `S3CHECK-FAIL ...` and is counted;
# - the LAST line is `S3CHECK-RESULT PASS|FAIL assertions=<n> failures=<n>`.
# The caller independently knows how many assertions this environment must produce and re-counts the
# OK lines itself, so neither a truncated log, a skipped bucket, an empty log, nor a lying result line
# can be read as success. (A CRD guard in this repo once counted a blank line as "1 CRD" and sailed
# past; that class of bug is designed out here rather than commented about.)
s3_probe_script() {
cat <<'PROBE'
set -u
umask 077
A=0; F=0
ok() { A=$((A+1)); printf 'S3CHECK-OK[%d] %s\n' "$A" "$*"; }
bad() { F=$((F+1)); printf 'S3CHECK-FAIL %s\n' "$*"; }
note() { printf 'S3CHECK-NOTE %s\n' "$*"; }
verdict() {
if [ "${F}" -eq 0 ] && [ "${A}" -gt 0 ]; then
printf 'S3CHECK-RESULT PASS assertions=%d failures=0\n' "${A}"; exit 0
fi
printf 'S3CHECK-RESULT FAIL assertions=%d failures=%d\n' "${A}" "${F}"; exit 1
}
for v in S3EP S3REGION S3TARGETS S3KEY CRED_MODE; do
eval "x=\${${v}-}"
[ -n "${x}" ] || { bad "required env ${v} is empty — the probe was launched wrong"; verdict; }
done
export AWS_EC2_METADATA_DISABLED=true
export AWS_PAGER=""
export AWS_DEFAULT_REGION="${S3REGION}"
export AWS_REGION="${S3REGION}"
export AWS_MAX_ATTEMPTS=2
export AWS_RETRY_MODE=standard
export AWS_CONFIG_FILE=/tmp/s3check-aws-config
# Path-style addressing, because both targets need it: MinIO on a bare IP has no virtual-host names,
# and R2's S3 API is documented path-style. Checksums OFF, mirroring `checksumAlgorithm: ""` on the
# Velero BSL — this run must exercise the same wire shape the real backup will use. Written to the
# CONFIG FILE as well as the environment so the setting holds even if one channel is ignored by a
# future CLI version.
{
echo '[default]'
echo 'request_checksum_calculation = when_required'
echo 'response_checksum_validation = when_required'
echo 's3 ='
echo ' addressing_style = path'
} > "${AWS_CONFIG_FILE}"
# ---- (1) the credential loads at all -------------------------------------------------
# NEVER print a credential value, in either `key: value` or `key=value` shape. Only booleans,
# byte counts and lengths leave this pod.
case "${CRED_MODE}" in
file)
cf="${AWS_SHARED_CREDENTIALS_FILE-}"
if [ -z "${cf}" ] || [ ! -s "${cf}" ]; then
bad "credential file '${cf:-<unset>}' is missing or empty — the Secret key did not mount"
elif ! grep -Eq '^[[:space:]]*\[default\][[:space:]]*$' "${cf}"; then
bad "credential file has no [default] profile. velero-plugin-for-aws reads an AWS INI file from
the 'cloud' key of velero-cloud-credentials; without [default] every backup fails at RUNTIME"
elif ! grep -Eq '^[[:space:]]*aws_access_key_id[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}" \
|| ! grep -Eq '^[[:space:]]*aws_secret_access_key[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}"; then
bad "credential file [default] lacks a non-empty aws_access_key_id / aws_secret_access_key"
else
ok "credential LOADS: AWS INI with [default] + both keys ($(wc -c < "${cf}") bytes; values never printed)"
fi
;;
env)
if [ -z "${AWS_ACCESS_KEY_ID-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
bad "AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY did not arrive from the Secret (missing ACCESS_KEY_ID / ACCESS_SECRET_KEY key?)"
else
ok "credential LOADS: access key id present (${#AWS_ACCESS_KEY_ID} chars; values never printed)"
fi
;;
*) bad "unknown CRED_MODE '${CRED_MODE}'" ;;
esac
[ "${F}" -eq 0 ] || verdict # no point talking to S3 with no credential
OUT=""; RC=0
awsx() { OUT="$(aws --endpoint-url "${S3EP}" "$@" 2>&1)"; RC=$?; return 0; }
detail() { printf '%s\n' "${OUT}" | head -n 12 | while IFS= read -r l; do printf 'S3CHECK-DETAIL %s\n' "${l}"; done; }
# Every failure is NAMED and given the fix. An unclassified failure is still a failure.
diagnose() { # <operation label>
local op="$1"
case "${OUT}" in
*InvalidArgument*|*checksum*|*Checksum*)
bad "${op}: 400 InvalidArgument — THE R2 CHECKSUM TRAP. velero-plugin-for-aws v1.14+ bundles an
AWS SDK that sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT, and
Cloudflare R2 REJECTS those headers with HTTP 400 InvalidArgument, so every Velero backup and
every kopia blob upload fails. FIX: checksumAlgorithm: \"\" must be present in the
backupStorageLocation config in platform/velero/helmrelease.yaml. This probe already disables
checksums, so seeing this here means the client sent one anyway — treat it as a hard blocker." ;;
*"Could not connect"*|*EndpointConnectionError*|*ConnectTimeoutError*|*"Failed to establish"*|\
*"Name or service not known"*|*"nodename nor servname"*|*"Temporary failure in name resolution"*|\
*"Connection reset"*|*"Read timeout"*)
bad "${op}: this POD cannot reach ${S3EP} (connect / DNS / timeout). Velero and CNPG would
install perfectly clean and then fail every backup at RUNTIME — a tier that looks healthy
while no backup exists. This is the reading that matters: the workstation's own reachability
is irrelevant (prod sits on the SOHO net, pfSense's WAN; the dev LAN's MinIO sits INSIDE
pfSense) and a host-side probe has already misled this project once. Fix pod egress/routing,
or give this environment its own endpoint." ;;
*NoSuchBucket*|*"(404)"*|*"Not Found"*)
bad "${op}: the bucket does not exist at this endpoint. Create it before enabling velero/CNPG
(R2: dashboard -> R2 -> Create bucket; see infra/BACKUP-R2.md)." ;;
*InvalidAccessKeyId*|*SignatureDoesNotMatch*|*"(401)"*|*Unauthorized*)
bad "${op}: the endpoint REJECTED these credentials. Wrong key, or a key belonging to a
different R2 account / MinIO. Re-encrypt the credential Secrets — infra/BACKUP-R2.md." ;;
*AccessDenied*|*"(403)"*|*Forbidden*)
bad "${op}: authenticated but FORBIDDEN. The token is not scoped to this bucket. An R2 API token
needs Object Read & Write on BOTH the velero and cnpg buckets." ;;
*) bad "${op}: unclassified failure (rc=${RC}). Treated as a FAILURE, never as unknown-so-fine." ;;
esac
detail
}
PAY=/tmp/s3check.payload
GOT=/tmp/s3check.got
head -c 4096 /dev/urandom | base64 > "${PAY}"
[ -s "${PAY}" ] || { bad "could not generate a probe payload inside the pod"; verdict; }
TRAP_DONE=0
for t in ${S3TARGETS}; do
B="${t%%:*}"; P="${t#*:}"
[ "${P}" = "${B}" ] && P=""
if [ -n "${P}" ]; then K="${P}/.s3check/${S3KEY}"; else K=".s3check/${S3KEY}"; fi
# ---- (2) the bucket exists and is LISTABLE. This is the operation Velero's BSL sync and Barman's
# WAL scan both perform; ListBuckets deliberately is NOT used, because neither consumer
# needs it and a correctly bucket-scoped R2 token may not grant it.
args=(s3api list-objects-v2 --bucket "${B}" --max-keys 1)
[ -n "${P}" ] && args+=(--prefix "${P}/")
awsx "${args[@]}"
if [ "${RC}" -eq 0 ]; then
ok "${B}: bucket EXISTS, AUTHENTICATES and is LISTABLE (prefix '${P:-<bucket root>}')"
else
diagnose "LIST ${B}"
continue # a short assertion count is how the caller learns this target was abandoned
fi
# ---- (3) PUT -> GET -> byte-compare -> DELETE. A real write, not a HEAD: HEAD on a bucket proves
# nothing about whether an object can be STORED (wrong scope, checksum rejection and
# read-only tokens all pass a HEAD and fail a backup).
awsx s3api put-object --bucket "${B}" --key "${K}" --body "${PAY}"
if [ "${RC}" -eq 0 ]; then ok "${B}: PUT ${K} accepted (genuine write path)"
else diagnose "PUT ${B}/${K}"; continue; fi
rm -f "${GOT}"
awsx s3api get-object --bucket "${B}" --key "${K}" "${GOT}"
if [ "${RC}" -eq 0 ]; then ok "${B}: GET ${K} returned an object"
else diagnose "GET ${B}/${K}"
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${K}" >/dev/null 2>&1 || true
continue; fi
sp="$(wc -c < "${PAY}")"; sg="$(wc -c < "${GOT}" 2>/dev/null || echo 0)"
hp="$(sha256sum < "${PAY}")"; hp="${hp%% *}"
hg="$(sha256sum < "${GOT}")"; hg="${hg%% *}"
if [ "${sp}" = "${sg}" ] && [ -n "${hp}" ] && [ "${hp}" = "${hg}" ]; then
ok "${B}: round trip is BYTE-IDENTICAL (${sp} bytes, sha256 matches)"
else
bad "${B}: round trip CORRUPTED — wrote ${sp} bytes (sha256 ${hp}), read back ${sg} bytes (sha256 ${hg})"
fi
awsx s3api delete-object --bucket "${B}" --key "${K}"
if [ "${RC}" -eq 0 ]; then ok "${B}: DELETE ${K} accepted"
else diagnose "DELETE ${B}/${K}"; fi
awsx s3api head-object --bucket "${B}" --key "${K}"
if [ "${RC}" -ne 0 ]; then ok "${B}: scratch object is GONE — the probe left nothing behind"
else bad "${B}: scratch object ${K} STILL EXISTS after DELETE. Remove it by hand; the probe leaked."; fi
# ---- (4) the R2 checksum trap, probed ON PURPOSE. Informational by design: the authored BSL sets
# checksumAlgorithm: "", so a rejection here does NOT mean backups are broken — it means
# that line is LOAD-BEARING. Reported either way so the operator learns which world this
# endpoint lives in before, not after, enabling velero.
if [ "${TRAP_DONE}" -eq 0 ]; then
TRAP_DONE=1
TK="${K}.checksum-trap-probe"
OUT="$(AWS_REQUEST_CHECKSUM_CALCULATION=when_supported \
aws --endpoint-url "${S3EP}" s3api put-object --bucket "${B}" --key "${TK}" --body "${PAY}" 2>&1)"; RC=$?
if [ "${RC}" -eq 0 ]; then
note "checksum probe: this endpoint ACCEPTS a checksum-bearing PUT — the R2 checksum trap is NOT
present here (expected for MinIO). checksumAlgorithm: \"\" stays harmless."
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${TK}" >/dev/null 2>&1 \
|| note "checksum probe: could NOT delete ${TK} — remove it by hand"
else
case "${OUT}" in
*InvalidArgument*)
note "checksum probe: R2 CHECKSUM TRAP CONFIRMED at this endpoint — a PUT carrying
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 is rejected with 400 InvalidArgument.
checksumAlgorithm: \"\" in platform/velero/helmrelease.yaml is LOAD-BEARING: delete it, or bump
velero-plugin-for-aws without re-checking it, and EVERY backup breaks." ;;
*) note "checksum probe: inconclusive (rc=${RC}) — neither confirms nor rules out the trap." ;;
esac
fi
fi
done
verdict
PROBE
}
# ---------------------------------------------------------------- run one probe pod
# <ns> <file|env> <secret> <space-separated bucket:prefix targets> <endpoint> <region>
s3_probe() {
local ns="$1" mode="$2" sec="$3" targets="$4" ep="$5" region="$6"
# $$ alone is NOT unique here: inside the dev container this script is frequently PID 1, so two runs
# would reuse the same pod name AND the same scratch object key. Mix in the epoch second.
local pod="s3check-$$-${S3CHECK_RUN_ID}-${mode}"
# What this environment MUST produce: 1 credential assertion + 6 per bucket (list, put, get,
# byte-compare, delete, delete-verified). Computed HERE, by the caller, from the target list — the
# pod's own claim is never the only source of the expected count.
local t n=0
for t in ${targets}; do n=$((n+1)); done
local min=$((1 + 6 * n))
local volsjson="" mountsjson="" creds=""
case "${mode}" in
file) # exactly how velero consumes it: the `cloud` key as an AWS INI credentials FILE
volsjson='"volumes":[{"name":"s3creds","secret":{"secretName":"'"${sec}"'","items":[{"key":"cloud","path":"credentials"}],"defaultMode":256}}],'
mountsjson='"volumeMounts":[{"name":"s3creds","mountPath":"/s3creds","readOnly":true}],'
creds='{"name":"AWS_SHARED_CREDENTIALS_FILE","value":"/s3creds/credentials"},{"name":"CRED_MODE","value":"file"}' ;;
env) # exactly how CNPG/barman consumes it: discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys
creds='{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_KEY_ID"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_SECRET_KEY"}}},{"name":"CRED_MODE","value":"env"}' ;;
*) die "internal: unknown probe mode '${mode}'" ;;
esac
local b64; b64="$(s3_probe_script | base64 | tr -d '\n')"
[ -n "${b64}" ] || die "internal: probe script encoded to nothing"
# egress-class label: harmless here (the velero namespace is not an app namespace) but required by
# the require-egress-class-label ClusterPolicy in any namespace labelled app-namespace=true, which
# the CNPG credential namespaces are. Without it the probe pod would be REJECTED by admission and
# the check would report "unknown" instead of a result.
local overrides
overrides='{"apiVersion":"v1","metadata":{"labels":{"app.kubernetes.io/name":"s3check","homelab.keithjackson.dev/egress-class":"internet"}},"spec":{"restartPolicy":"Never","automountServiceAccountToken":false,'"${volsjson}"'"containers":[{"name":"s3check","image":"'"${S3CHECK_IMAGE}"'","imagePullPolicy":"IfNotPresent","command":["/bin/bash","-c","printf %s \"$PROBE_B64\" | base64 -d > /tmp/probe.sh && exec bash /tmp/probe.sh"],'"${mountsjson}"'"env":['"${creds}"',{"name":"PROBE_B64","value":"'"${b64}"'"},{"name":"S3EP","value":"'"${ep}"'"},{"name":"S3REGION","value":"'"${region}"'"},{"name":"S3TARGETS","value":"'"${targets}"'"},{"name":"S3KEY","value":"'"${pod}"'.probe"},{"name":"AWS_REQUEST_CHECKSUM_CALCULATION","value":"when_required"},{"name":"AWS_RESPONSE_CHECKSUM_VALIDATION","value":"when_required"}],"resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"memory":"256Mi"}}}]}}'
say "probe: ns=${ns} secret=${sec} (${mode}) targets='${targets}' → ${min} assertions required"
kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found >/dev/null 2>&1 || true
S3CHECK_PODS+=("${ns}/${pod}")
kubectl -n "${ns}" run "${pod}" --restart=Never --image="${S3CHECK_IMAGE}" \
--overrides="${overrides}" >/dev/null 2>&1 \
|| die "could NOT create the probe pod ${ns}/${pod} (admission policy? RBAC? quota?).
That is a FAILURE, not an unknown: nothing about ${ep} has been proven."
local i=0 phase=""
while [ "${i}" -lt 300 ]; do
phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
case "${phase}" in Succeeded|Failed) break ;; esac
sleep 5; i=$((i+5))
done
case "${phase}" in
Succeeded|Failed) ;;
*) local reason
reason="$(kubectl -n "${ns}" get pod "${pod}" \
-o jsonpath='{.status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)"
die "probe pod ${ns}/${pod} did not finish in ${i}s (phase=${phase:-<none>}, waiting=${reason:-none}).
NOT a pass. If waiting=ErrImagePull/ImagePullBackOff this cluster cannot pull ${S3CHECK_IMAGE}." ;;
esac
local log; log="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
[ -n "${log}" ] || die "probe pod ${ns}/${pod} produced NO output (phase=${phase}). Failing closed —
an empty log is the shape of a probe that never ran, not of a healthy backup target."
printf '%s\n' "${log}" | while IFS= read -r l; do printf ' %s\n' "${l}"; done
# ---- the four independent accounting checks that make a silent no-op impossible ----
local okcount failcount resline nres verd a f
okcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-OK\[' || true)"
failcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-FAIL' || true)"
nres="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-RESULT ' || true)"
[ "${nres}" = "1" ] || die "expected exactly ONE S3CHECK-RESULT line from ${ns}/${pod}, found ${nres}.
A missing verdict line means the probe died mid-way; it is never a pass."
resline="$(printf '%s\n' "${log}" | grep '^S3CHECK-RESULT ')"
read -r _ verd a f <<<"${resline}"
a="${a#assertions=}"; f="${f#failures=}"
case "${a}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
case "${f}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
[ "${verd}" = "PASS" ] || die "S3 ROUND TRIP FAILED against ${ep} (${ns}/${pod}: ${f} failure(s)).
Read the S3CHECK-FAIL lines above — each one names the cause and the fix. Do NOT enable velero
or CNPG backups against this endpoint until they are all gone."
[ "${f}" = "0" ] || die "${ns}/${pod} claimed PASS while reporting ${f} failures — refusing to believe it."
[ "${failcount}" = "0" ] || die "${ns}/${pod} claimed PASS but the log carries ${failcount} S3CHECK-FAIL line(s)."
[ "${a}" -ge "${min}" ] || die "${ns}/${pod} claimed PASS with only ${a} assertions; ${min} are required
(1 credential + 6 per bucket). A short count means steps were SKIPPED — exactly the
degrade-to-a-successful-no-op this check exists to prevent."
[ "${okcount}" = "${a}" ] || die "assertion accounting mismatch for ${ns}/${pod}: ${okcount} S3CHECK-OK
lines in the log vs a claimed ${a}. Refusing to pass on a self-report that does not add up."
[ "${phase}" = "Succeeded" ] || die "${ns}/${pod} printed PASS but the pod ended ${phase}."
info "PROVEN: ${a} assertions, 0 failures, pod ${phase}, no object left behind."
}
# ============================================================================ steps
step_preflight() {
say "PREFLIGHT (read-only)"
info "nodes Ready: $(kubectl get nodes --no-headers | awk '$2=="Ready"' | wc -l)/$(kubectl get nodes --no-headers | wc -l)"
info "flux controllers Running: $(kubectl -n flux-system get pods --no-headers | awk '$3=="Running"' | wc -l)"
# Count BOTH groups the Kyverno chart ships. Grepping only 'kyverno.io' undercounts by 2 — the
# policy-report types are in the wgpolicyk8s.io group — which reads as "CRDs are missing" when the
# pre-apply in fact succeeded. Expect 22 for kyverno 3.9.x.
info "kyverno CRDs present: $(kubectl get crd --no-headers 2>/dev/null | grep -cE 'kyverno\.io|wgpolicyk8s\.io' || true) (expect 22 for kyverno 3.9.x)"
local base owner filter s3 acme
base="$(cvar BASE_DOMAIN)"; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
filter="$(cvar EXTERNALDNS_DOMAIN_FILTER)"; s3="$(cvar S3_BACKUP_ENDPOINT)"; acme="$(cvar ACME_CLUSTER_ISSUER)"
say "cluster-vars guards"
info "BASE_DOMAIN = ${base}"
info "EXTERNALDNS_TXT_OWNER_ID = ${owner}"
info "EXTERNALDNS_DOMAIN_FILTER = ${filter}"
info "ACME_CLUSTER_ISSUER = ${acme}"
info "S3_BACKUP_ENDPOINT = ${s3}"
# THE GUARD THAT MATTERS. Two clusters sharing one Cloudflare zone are isolated by txtOwnerId and
# by serving disjoint hostnames — NOT by domainFilters (finding F-DNS1: a subdomain filter drops the
# parent zone and publishes nothing). Both clusters inheriting the chart default `default` is the
# single root cause of the 2026-09-04 incident, in which prod deleted every one of dev's records.
# preflight is a DIAGNOSTIC: it must always finish and report. The only thing it refuses over is the
# target guard, because that one means you are pointed at the wrong cluster. Everything else is
# reported and warned about here, and REFUSED on the mutating path (step_externaldns) where acting
# on a bad value is what actually causes harm. Dying here instead hid the rest of dev's health
# report behind a condition that was, for dev, simply the long-standing status quo.
if [ -z "${owner}" ]; then
warn "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns falls back to the chart default 'default'.
Safe only if this is the sole cluster publishing into the zone. 'externaldns' will refuse."
elif [ "${owner}" = "unset-see-cluster-vars" ]; then
warn "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply.
'externaldns' will refuse until a real per-environment value is set."
elif [ "${owner}" = "default" ]; then
# NOT fatal here, because `default` is DEV's legitimate, established owner id — its live records
# are registered under it and cannot be renamed without orphaning them. Dying on it would make
# this script refuse to run against dev at all, which would be an environment fork (R-ENV-02) in
# a script whose whole claim is that it has none. preflight only READS, so it warns; the
# MUTATING path (step_externaldns) still refuses outright.
warn "EXTERNALDNS_TXT_OWNER_ID is 'default' — the Helm chart's default value.
That is correct for DEV (its records are registered under it) and WRONG for any second cluster
sharing this zone: two clusters both defaulting to 'default' is the exact root cause of the
2026-09-04 incident, in which prod treated dev's TXT registry entries as its own and
policy: sync deleted the records behind them."
else
info "owner id is distinct and non-default: OK"
fi
if s3_is_placeholder "${s3}"; then
warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
velero will be SKIPPED by the resume step. apps-wave-0 will still apply (the token IS defined,
so strict substitution succeeds) but anything writing to that endpoint fails at RUNTIME, not at
apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required."
else
info "S3 backup endpoint set: OK"
fi
# COLD-START CRD ORDERING. Flux server-side dry-runs EVERY object in a Kustomization before applying
# any of them, so a single unknown `kind` aborts the whole tier. platform-config applies the CDI CR
# *and* CDI StorageProfiles in the same pass — but the StorageProfile CRD does not exist until the
# cdi-operator has reconciled that CR. On an established cluster the CRDs are long since there and
# this is invisible; on a COLD one it is a deadlock, and it is the same class of bug as Kyverno's
# 5.7 MiB CRD bundle. Report it here so `resume` does not burn 15 minutes discovering it.
say "CRDs that later tiers apply resources against"
local crd missing=0
for crd in storageprofiles.cdi.kubevirt.io cdis.cdi.kubevirt.io kubevirts.kubevirt.io \
volumesnapshotclasses.snapshot.storage.k8s.io; do
if kubectl get crd "${crd}" >/dev/null 2>&1; then
printf ' %-52s present\n' "${crd}"
else
printf ' \033[33m%-52s MISSING\033[0m\n' "${crd}"; missing=1
fi
done
if [ "${missing}" = "1" ]; then
warn "a MISSING crd above will fail platform-config's dry-run and block every tier below it.
CDI's own CRDs (StorageProfile, DataVolume, ...) are created by cdi-operator only AFTER it
reconciles the CDI CR. Check that the CDI CR is applied and cdi-operator is Available; if the
CRD appeared only recently this can also be a stale discovery cache in kustomize-controller,
which a forced reconcile clears."
kubectl -n cdi get cdi,deploy 2>/dev/null | head -12 || info "(no cdi namespace yet)"
fi
say "tiers"
local n
for n in flux-system cilium "${TIER_ORDER[@]}" velero; do
printf ' %-16s suspend=%-6s ready=%-6s %s\n' "$n" "$(k_susp "$n")" "$(k_ready "$n")" "$(k_msg "$n" | head -c 90)"
done
say "HelmReleases not Ready"
kubectl get helmrelease -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message \
--no-headers 2>/dev/null | grep -v ' True ' || info "(all Ready)"
# SUSPENDED releases are invisible above: a suspended HelmRelease keeps whatever Ready condition it
# last had, so a STALE `Ready=True` hides the fact that Flux is not managing it at all. external-dns
# is deliberately in this state until `standup.sh externaldns` runs.
say "HelmReleases SUSPENDED (Flux is not reconciling these — a stale Ready=True means nothing)"
kubectl get helmrelease -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend --no-headers 2>/dev/null \
| awk '$3=="true"' || true
}
step_crds() {
say "CRD PRE-APPLY (idempotent)"
# Charts whose CRD bundle exceeds kubectl's 256 KiB annotation limit set crds.install=false, so
# nothing installs them and `platform` can never go Ready. See bootstrap/flux/preapply-crds.sh.
bash "${script_dir}/flux/preapply-crds.sh"
# ---------------------------------------------------------------- the CDI cold-start deadlock
# SAME CLASS OF BUG, DIFFERENT OPERATOR. `platform-config` applies the CDI CR *and*
# platform/kubevirt/cdi-storageprofiles.yaml in ONE pass. But StorageProfile's CRD does not exist
# until cdi-operator has reconciled that CR — and Flux server-side dry-runs EVERY object before
# applying ANY of them, so the unknown kind aborts the whole tier, the CR is never applied, the CRD
# is never created, and platform-config can never converge. A true deadlock, invisible on an
# established cluster (the CRDs are long since there) and fatal on a cold one.
#
# Breaking it needs exactly one thing: get the CDI CR in. Then cdi-operator creates the remaining
# CDI CRDs and platform-config's next reconcile dry-runs clean. We apply ONLY the CR — the minimum
# — and leave every other object to Flux. The CR carries no ${} tokens, so no substitution is
# needed and applying it here cannot diverge from what Flux would render.
local cdi_cr="${repo_root}/platform/kubevirt/cdi-cr.yaml"
if kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1; then
info "storageprofiles CRD already present — CDI is past its cold start"
elif [ ! -f "${cdi_cr}" ]; then
warn "expected ${cdi_cr} and it is missing — skipping the CDI unblock"
else
say "CDI cold-start unblock: applying the CDI CR so cdi-operator can create its CRDs"
kubectl get ns cdi >/dev/null 2>&1 || die "namespace 'cdi' absent — the platform tier has not applied the CDI operator yet. Run 'resume' far enough to make platform Ready first."
kubectl apply --server-side --force-conflicts -f "${cdi_cr}"
local i=0
while [ "$i" -lt 300 ]; do
kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 && break
sleep 10; i=$((i+10))
done
kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 \
|| die "cdi-operator did not create storageprofiles.cdi.kubevirt.io within ${i}s.
Inspect: kubectl -n cdi get cdi,pods,deploy — platform-config stays blocked until it exists."
kubectl wait --for=condition=Established --timeout=60s crd/storageprofiles.cdi.kubevirt.io >/dev/null
info "storageprofiles.cdi.kubevirt.io created and Established (${i}s)"
fi
}
step_unwedge() {
say "UNWEDGE STUCK HELM RELEASES"
# A Helm release left in uninstalling/pending-* cannot be advanced by helm-controller: it reports
# StateError ("unable to determine state for release with status 'uninstalling'") forever, which
# keeps `platform` (wait: true) not-Ready and every tier below it blocked.
#
# How prod got here: Kyverno's install timed out because its pods were crash-looping with no CRDs;
# install.remediation.retries:3 then fired an uninstall, and that uninstall timed out too.
#
# Pre-applied CRDs live OUTSIDE the Helm release and SURVIVE this — that is precisely why the
# pre-apply is a separate step and must run before this one.
local found=0 ns rel st
while read -r ns rel st; do
[ -n "${ns:-}" ] || continue
found=1
warn "${ns}/${rel} is '${st}'"
[ "${CONFIRM_DESTRUCTIVE}" = "1" ] || {
info "DRY RUN. This DELETES the release's Kubernetes objects and lets Flux install cold."
info "Re-run with --confirm-destructive to act."
continue
}
say "clearing ${ns}/${rel}"
# --no-hooks and --wait=false: the WAIT is what hung the original remediation, so do not repeat it.
helm uninstall "${rel}" -n "${ns}" --no-hooks --wait=false || warn "helm uninstall returned non-zero; continuing to check state"
if helm list -n "${ns}" --all -q 2>/dev/null | grep -qx "${rel}"; then
warn "${ns}/${rel} still present after uninstall — inspect manually, NOT deleting release secrets by hand."
else
info "${ns}/${rel} cleared"
flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile of ${ns}/${rel} did not complete in 10m"
fi
done < <(
for ns in $(kubectl get helmrelease -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u); do
helm list -n "${ns}" --all -o json 2>/dev/null \
| yq -r '.[] | select(.status | test("uninstalling|pending-install|pending-upgrade|pending-rollback")) | "'"${ns}"' " + .name + " " + .status' 2>/dev/null || true
done
)
[ "${found}" = "1" ] || info "no wedged releases found."
}
step_externaldns() {
say "EXTERNAL-DNS — SAFE UPGRADE (prove the owner id BEFORE the pod can run)"
# THE HAZARD: a Helm upgrade sets replicas back to the chart default (1) and applies the new args in
# ONE operation. There is no window in which to check the result first. If --txt-owner-id is missing
# the pod starts as owner `default`, claims the other cluster's TXT registry entries, and under
# `policy: sync` DELETES the records behind them. That is the 2026-09-04 incident.
#
# THE FIX IS TO VERIFY IN ADVANCE, NOT AFTER: render the chart with the LIVE HelmRelease values and
# assert the flag appears. Rendering is deterministic and touches nothing, so this converts a race
# into a precondition.
local ns=external-dns rel=external-dns
local owner chart ver repo vals
owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
[ -n "${owner}" ] && [ "${owner}" != "default" ] && [ "${owner}" != "unset-see-cluster-vars" ] \
|| die "refusing: EXTERNALDNS_TXT_OWNER_ID is '${owner}'"
chart="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.chart}')"
ver="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.version}')"
repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' platform/externaldns/helmrepository.yaml | tr -d '"')"
[ -n "${repo}" ] || die "could not derive the external-dns chart repo url"
vals="${SNAP_DIR}/ed-values.yaml"
kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.values}' | yq -P > "${vals}"
info "chart=${chart} version=${ver} repo=${repo}"
info "live spec.values.txtOwnerId = $(yq '.txtOwnerId' "${vals}")"
say "rendering the chart with the LIVE values to prove the flag"
local rendered="${SNAP_DIR}/ed-rendered.yaml"
helm template "${rel}" "${chart}" --repo "${repo}" --version "${ver}" -n "${ns}" -f "${vals}" > "${rendered}"
local args; args="$(yq -r 'select(.kind == "Deployment") | .spec.template.spec.containers[0].args[]' "${rendered}")"
printf '%s\n' "${args}" | sed 's/^/ /'
printf '%s\n' "${args}" | grep -qx -- "--txt-owner-id=${owner}" \
|| die "the render does NOT contain --txt-owner-id=${owner}.
An upgrade would start external-dns as owner 'default' and delete the other cluster's records.
REFUSING. Fix the chart values key before retrying."
info "PROVEN: the upgrade will carry --txt-owner-id=${owner}"
say "snapshotting the Cloudflare zone (baseline for the post-check)"
dns_snapshot "${SNAP_DIR}/dns-before.txt"
say "resuming + forcing the upgrade"
kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.suspend}' | grep -q true \
&& flux resume helmrelease "${rel}" -n "${ns}" || info "(HelmRelease was not suspended)"
flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile did not complete in 10m"
say "verifying the LIVE deployment args"
local live; live="$(kubectl -n "${ns}" get deploy "${rel}" -o jsonpath='{.spec.template.spec.containers[0].args}' 2>/dev/null || true)"
printf ' %s\n' "${live}"
printf '%s' "${live}" | grep -q -- "--txt-owner-id=${owner}" \
|| { kubectl -n "${ns}" scale deploy "${rel}" --replicas=0 || true
die "live args LACK --txt-owner-id=${owner}. Scaled to 0. Investigate before scaling up."; }
info "live deployment carries --txt-owner-id=${owner}"
say "post-check: did anything in the shared zone change?"
sleep 45 # external-dns --interval=1m; give it a full cycle to act before judging it safe
dns_snapshot "${SNAP_DIR}/dns-after.txt"
dns_diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-after.txt" \
|| warn "records changed — expected ONLY additions under this cluster's own BASE_DOMAIN ($(cvar BASE_DOMAIN)).
Any change to a name outside it, or to a TXT entry owned by another cluster, is a REGRESSION."
}
step_s3check() {
say "S3 BACKUP TARGET — PROVE A REAL ROUND TRIP FROM INSIDE THE CLUSTER"
# Velero and CNPG run in-cluster, so pod egress is the only reachability that counts: prod sits on
# the SOHO network (pfSense's WAN) while the dev LAN's MinIO sits INSIDE pfSense, so host and pod
# egress differ — and a healthy host-side reading has already misled this project once.
#
# WHAT THIS PROVES (and it fails closed on every one of them):
# 1. the committed credential loads and AUTHENTICATES, in the exact shape each consumer reads it
# (velero: an AWS INI file from the `cloud` key; CNPG: discrete ACCESS_KEY_ID/ACCESS_SECRET_KEY);
# 2. each required bucket EXISTS and is LISTABLE under this environment's own prefix;
# 3. PUT -> GET -> byte-compare -> DELETE of a throwaway object under that prefix — the genuine
# write path, not a HEAD (a HEAD passes with a read-only token and with R2 rejecting writes);
# 4. whether the R2 CHECKSUM TRAP is present at this endpoint, named explicitly either way.
#
# WHAT IT STILL DOES NOT PROVE: that Velero's kopia uploader and CNPG's barman-cloud themselves work
# against this target. Both are UNPROVEN against R2 in this repo. See infra/BACKUP-R2.md.
local ep live region
live="$(cvar S3_BACKUP_ENDPOINT)"
info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then
# Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed
# backup target works from this cluster's pods BEFORE authoring it into cluster-vars — otherwise
# the first evidence is silently-missing backups.
ep="${S3_ENDPOINT_OVERRIDE}"
info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}"
else
ep="${live}"
fi
if s3_is_placeholder "${ep}"; then
die "no usable endpoint to probe (endpoint='${ep:-<unset>}').
cluster-vars still carries the loud placeholder. Pass --s3-endpoint https://<account-id>.r2.cloudflarestorage.com
to test a CANDIDATE before authoring it. See infra/BACKUP-R2.md."
fi
s3_safe_token S3_BACKUP_ENDPOINT "${ep}"
case "${ep}" in
http://*|https://*) ;;
*) die "endpoint '${ep}' carries no http:// or https:// scheme — the AWS SDK needs one." ;;
esac
# Region is a knob, not a constant: MinIO ignores it, R2 requires exactly `auto`, and an EMPTY value
# in cluster-vars silently builds a BSL with no region.
region="$(cvar S3_BACKUP_REGION)"
case "${ep}" in
*.r2.cloudflarestorage.com*)
if [ "${region}" != "auto" ]; then
warn "endpoint is Cloudflare R2 but cluster-vars S3_BACKUP_REGION='${region:-<unset>}'. R2 requires
exactly 'auto'. Probing with 'auto' anyway; FIX clusters/base/cluster-vars.yaml before enabling
velero, or the BackupStorageLocation is built with the wrong region."
fi
region=auto ;;
*)
if [ -z "${region}" ]; then
region="us-east-1"
warn "cluster-vars S3_BACKUP_REGION is EMPTY on this cluster. Probing with 'us-east-1' (MinIO
ignores it) — but the BSL would be rendered with region: \"\". Set it explicitly."
fi ;;
esac
s3_safe_token S3_BACKUP_REGION "${region}"
# Buckets come from the AUTHORED manifests + cluster-vars, never from a hardcoded name here: the
# whole point is to probe what this environment will really write to (R-ENV-02, no dev/prod fork).
local vbucket vprefix cpath cbucket cprefix rest
vbucket="$(yq -r '.spec.values.configuration.backupStorageLocation[0].bucket' \
platform/velero/helmrelease.yaml 2>/dev/null | tr -d '"')"
[ -n "${vbucket}" ] && [ "${vbucket}" != "null" ] \
|| die "could not read the Velero bucket name from platform/velero/helmrelease.yaml"
vprefix="$(cvar VELERO_BACKUP_PREFIX)"
cpath="$(cvar CNPG_BACKUP_PATH)"
if [ -z "${cpath}" ]; then
warn "cluster-vars CNPG_BACKUP_PATH is EMPTY on this cluster. barmanObjectStore.destinationPath would
substitute to the empty string and EVERY WAL archive would fail at runtime. Probing the historical
default s3://cnpg so the round trip is still proven — but SET THIS VALUE."
cpath="s3://cnpg"
fi
case "${cpath}" in s3://*) ;; *) die "CNPG_BACKUP_PATH='${cpath}' is not an s3:// URL." ;; esac
rest="${cpath#s3://}"; cbucket="${rest%%/*}"
if [ "${rest}" = "${cbucket}" ]; then cprefix=""; else cprefix="${rest#*/}"; fi
cprefix="${cprefix%/}"
[ -n "${cbucket}" ] || die "CNPG_BACKUP_PATH='${cpath}' names no bucket."
s3_safe_token velero-bucket "${vbucket}"; s3_safe_token VELERO_BACKUP_PREFIX "${vprefix}"
s3_safe_token cnpg-bucket "${cbucket}"; s3_safe_token CNPG_BACKUP_PATH-prefix "${cprefix}"
say "what is about to be proven"
info "endpoint ${ep}"
info "region ${region}"
info "velero bucket '${vbucket}' prefix '${vprefix:-<bucket root>}' (authored bucket + VELERO_BACKUP_PREFIX)"
info "cnpg bucket '${cbucket}' prefix '${cprefix:-<bucket root>}' (from CNPG_BACKUP_PATH)"
info "scratch key <prefix>/.s3check/<pod>.probe — written, read back, byte-compared, DELETED"
local ran=0 keys
# (a) Velero's own credential, from its own namespace, mounted the way the aws plugin reads it.
if kubectl get ns velero >/dev/null 2>&1 \
&& kubectl -n velero get secret velero-cloud-credentials >/dev/null 2>&1; then
# List KEY NAMES only. Never pipe or print a value (a careless redaction has leaked a password
# into a transcript on this project already). go-template, not jsonpath: kubectl's jsonpath
# cannot range over a map's keys (`range $k,$v :=` is rejected, U+002C) and returns EMPTY, which
# silently looked like "the key is missing".
keys="$(kubectl -n velero get secret velero-cloud-credentials \
-o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' 2>/dev/null || true)"
printf '%s\n' "${keys}" | grep -qx 'cloud' \
|| die "Secret velero/velero-cloud-credentials carries no 'cloud' key (keys: $(printf '%s' "${keys}" | tr '\n' ' ')).
velero-plugin-for-aws reads exactly that key as an AWS INI file; without it every backup fails
at RUNTIME while the HelmRelease reports Ready."
s3_probe velero file velero-cloud-credentials "${vbucket}:${vprefix}" "${ep}" "${region}"
ran=$((ran+1))
else
warn "no Secret velero/velero-cloud-credentials on this cluster — Velero's own credential could NOT
be proven. That is expected before the velero tier has ever reconciled here; it is not a pass."
fi
# (b) A CNPG/barman credential, discovered rather than hardcoded (R-ENV-02: no per-environment list
# of app names in this script). Any one of them proves the key + the cnpg bucket + the discrete
# ACCESS_KEY_ID/ACCESS_SECRET_KEY shape; they are all kept in sync by design.
local pick cns csec
pick="$(kubectl get secret -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' 2>/dev/null \
| grep -- '-db-backup-credentials$' | head -n 1 || true)"
if [ -n "${pick}" ]; then
cns="${pick%%/*}"; csec="${pick#*/}"
s3_probe "${cns}" env "${csec}" "${cbucket}:${cprefix}" "${ep}" "${region}"
ran=$((ran+1))
else
warn "no *-db-backup-credentials Secret found in any namespace — the CNPG/barman credential could
NOT be proven."
fi
[ "${ran}" -gt 0 ] || die "NO backup credential Secret exists on this cluster, so NOTHING could be
proven about ${ep}. That is a FAILURE, not a skip: a green s3check must never mean 'I found
nothing to check'."
say "S3 ROUND TRIP PROVEN from inside the cluster: ${ep} (${ran} credential path(s))"
info "STILL UNPROVEN by this step: Velero's kopia uploader and CNPG barman-cloud against this target,"
info "and a RESTORE. A backup that has never been restored is not a backup — see infra/BACKUP-R2.md."
}
step_resume() {
say "RESUME TIERS (dependency order)"
local n s3; s3="$(cvar S3_BACKUP_ENDPOINT)"
for n in "${TIER_ORDER[@]}"; do
if [ "$(k_susp "$n")" = "true" ]; then
info "resuming $n"; flux resume kustomization "$n" --timeout=1s >/dev/null 2>&1 || true
else
info "$n already unsuspended"
fi
wait_ready "$n" 900 || { warn "stopping at $n — a partial, understood bring-up beats a forced green."; return 1; }
done
# The placeholder test MUST be substring-based. It was `TBD-*` (a PREFIX match) while the placeholder
# was `TBD-PROD-…`; the R2 move changed it to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`,
# which does not START with TBD- — so this case arm silently stopped matching and `resume` would have
# RESUMED velero onto a placeholder hostname on prod. That is the precise failure this guard exists
# to prevent, reintroduced by a value change in another file. s3_is_placeholder matches anywhere.
if s3_is_placeholder "${s3}"; then
warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
Resuming it would create a BackupStorageLocation that can never reach a bucket, i.e. a tier
that reports Ready-ish while backups silently do not exist. Set the real endpoint first, prove
it with \`s3check\`, and only then resume (infra/BACKUP-R2.md)."
else
info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
wait_ready velero 900 || warn "velero did not converge"
fi
}
step_verify() {
say "VERIFY"
local base; base="$(cvar BASE_DOMAIN)"
info "BASE_DOMAIN = ${base}"
say "kustomizations"
kubectl get kustomization -A \
-o custom-columns=NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status --no-headers
say "certificates (must be Ready before any hostname is trusted)"
kubectl get certificate -A --no-headers 2>/dev/null || info "(none yet)"
say "pods not Running/Completed"
kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed"' || true
info "total pods: $(kubectl get pods -A --no-headers | wc -l)"
say "gateway / LoadBalancer VIPs"
kubectl get svc -A --field-selector spec.type=LoadBalancer --no-headers 2>/dev/null || true
say "published DNS for this cluster's own domain"
local owner; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
dns_snapshot "${SNAP_DIR}/dns-verify.txt"
info "records under ${base}: $(grep -c "${base}" "${SNAP_DIR}/dns-verify.txt" || true)"
# Every TXT registry entry this cluster publishes must carry OUR owner id. One reading another
# cluster's id means we have claimed its records and `policy: sync` will delete them next cycle.
local foreign
foreign="$(grep "${base}" "${SNAP_DIR}/dns-verify.txt" | grep 'heritage=external-dns' \
| grep -v "external-dns/owner=${owner}" || true)"
if [ -n "${foreign}" ]; then
warn "TXT entries under ${base} NOT owned by '${owner}' — this cluster has claimed records it does not own:"
printf '%s\n' "${foreign}" | sed 's/^/ /'
else
info "every TXT registry entry under ${base} reads owner=${owner}: OK"
fi
# THE REGRESSION CHECK. Additions under our own BASE_DOMAIN are expected. ANY change to a name
# outside it means we have touched the other cluster's zone — the 2026-09-04 incident.
if [ -f "${SNAP_DIR}/dns-before.txt" ]; then
say "regression check against the pre-standup baseline"
local outside
outside="$(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" \
| grep -E '^[<>]' | grep -v "${base}" || true)"
if [ -n "${outside}" ]; then
warn "RECORDS OUTSIDE ${base} CHANGED — investigate before doing anything else:"
printf '%s\n' "${outside}" | sed 's/^/ /'
else
info "no record outside ${base} was added, changed or removed: OK"
info "added under ${base}: $(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" | grep -c '^>' || true)"
fi
else
warn "no ${SNAP_DIR}/dns-before.txt baseline — cannot prove the other cluster's records are untouched.
Take one BEFORE the next standup so this check can run."
fi
}
# ============================================================================ dispatch
for s in "${STEPS[@]}"; do
case "$s" in
preflight) step_preflight ;;
s3check) step_s3check ;;
crds) step_crds ;;
unwedge) step_unwedge ;;
externaldns) step_externaldns ;;
resume) step_resume ;;
verify) step_verify ;;
all) step_preflight; step_crds; step_unwedge; step_externaldns; step_resume; step_verify ;;
*) die "unknown step '$s' (preflight crds unwedge externaldns s3check resume verify all)" ;;
esac
done
say "done: ${STEPS[*]}"
-28
View File
@@ -180,34 +180,6 @@ dependencies: [] # default-deny: reaches only DNS; git clients re
ingressRoutes:
- host: gitea.${BASE_DOMAIN}
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
#
# Prod's Flux source is THIS SERVICE, not a public hostname:
# clusters/prod/kustomization.yaml -> GitRepository.spec.url =
# http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
# file documents the other half of this pair at length; the two must be read together.
#
# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
# source is down — source down -> routes never rename -> source stays down. It was broken on
# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
# cluster object that existed OUTSIDE git. This declaration renders that object, so one
# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
inClusterClients:
# `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
# equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
# near-duplicate policy beside it, and no window in which prod's own source has no allow.
- name: flux
namespace: flux-system # matched by NAMESPACE alone — the flux controllers' pod labels belong
# to upstream's manifests, not to this platform's contract
port: 3000 # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
auth:
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
groups:
-20
View File
@@ -23,17 +23,6 @@ storage:
- name: config
size: 2Gi
class: longhorn # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
ingressRoutes:
- host: jackett.${BASE_DOMAIN}
@@ -42,15 +31,6 @@ auth:
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in an authorized group (R-IDP-06)
# platform-admins is a SUPERSET, not a widening. --allowed-group is repeatable and OR-ed, so
# listing both means "media consumers OR platform operators". Without it the media UIs are
# reachable by NOBODY the moment media-users is empty: on prod the realm's only user is the
# operator, who is in platform-admins only, so jackett/qbittorrent/radarr/sonarr answered an
# immediate 403 at the oauth2-proxy callback (no login prompt, because the Keycloak SSO session
# already existed) while pihole/crafty — which already gate on platform-admins — worked.
# This grants no real privilege: platform-admins already maps to cluster-admin via
# k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=0.22.0" # stay on the Jackett 0.22.x stable line
-10
View File
@@ -31,15 +31,5 @@ observability:
metrics: true
slis:
- up # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
# WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
# `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
# imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
# is the catalog's declaration of intent for it. Without this override the rendered SLI would
# read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
# Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
# statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
livenessWorkload:
kind: StatefulSet
name: keycloak
backup:
policy: daily-offcluster # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)
-38
View File
@@ -223,44 +223,6 @@ storage:
mountPath: /var/www/html # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
# The whole html dir is persisted so config.php + user data survive a
# rebuild (the default /data/<name> would strand them on ephemeral rootfs).
# THE LEGACY NEXTCLOUD DATASET (Storage/nextcloud, 270 G) — MOUNTED READ-ONLY, DELIBERATELY, AND
# NOT AS THIS INSTANCE'S DATA DIRECTORY. Read this before changing either decision.
#
# WHAT IS ACTUALLY IN THERE (measured 2026-09-07, not assumed):
# /mnt/Storage/nextcloud/nextcloud/config 103 M
# /mnt/Storage/nextcloud/nextcloud/config_old 3.4 G
# /mnt/Storage/nextcloud/nextcloud/config_backup 267 G <- 99% of the dataset
# There is NO `data/` directory. This is the OLD, non-k3s Nextcloud's configuration plus a backup
# blob — an ARCHIVE, not a live user-data tree.
#
# WHY READ-ONLY. Two independent reasons, either sufficient:
# 1. The old instance is STILL LIVE on it. The export is currently shared rw to 192.168.1.6 and
# 192.168.1.5, and the operator has said the existing non-k3s prod must remain active. Two
# Nextcloud instances writing one instance's config directory, each with its own database and
# therefore its own oc_filecache, is a corruption scenario with no clean recovery.
# 2. Nothing here needs writing. An archive is read.
# "Do not grant more than needed" therefore lands on `ro`. If the operator's intent is in fact a
# MIGRATION (read this data in, then decommission the old instance), that is a different and
# larger change than a mount, and it should be a deliberate one — flip `readOnly` and update the
# export at that point, not before. Enforced by the PV's nfs.readOnly, so it is a real `ro` mount.
#
# WHY NOT /var/www/html/data. That path is the LONGHORN PVC above and is this instance's own
# state; mounting a foreign tree into it would have Nextcloud scan and index an archive it does
# not own. A neutral path keeps the two completely separate. To expose it to users, add it as a
# Nextcloud External Storage (Local) mount pointing at this path — an in-app, per-user decision.
#
# subPath skips the redundant single-child directory: the dataset root holds exactly one entry,
# `nextcloud/`, so mounting that subtree puts config/ config_old/ config_backup/ at the top.
#
# UID: nothing to do. The tree is 0755 owned 1000:3000 and the mount is read-only, so www-data
# (uid 33) can read and traverse it. (fsGroup would be a no-op on an NFS volume in any case.)
- name: legacy-archive
class: bulk
export: /mnt/Storage/nextcloud # the ZFS dataset Storage/nextcloud, on ${BULK_NFS_SERVER}
subPath: nextcloud
mountPath: /mnt/legacy-nextcloud
readOnly: true
size: 270Gi # ADVISORY ONLY on a static NFS PV — the dataset's used size.
dependencies:
- to: nextcloud-db # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
-87
View File
@@ -13,93 +13,6 @@ storage:
size: 2Gi
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
# ------------------------------------------------------------------------------------------------
# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
# ------------------------------------------------------------------------------------------------
# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
#
# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
#
# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
# configured through the UI is lost on the next reschedule — which is exactly what happened to the
# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
env:
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
- name: FTLCONF_dns_listeningMode
value: "ALL"
# Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
# router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
# answers, and forwarding to a device that may forward back here invites a loop.
- name: FTLCONF_dns_upstreams
value: "8.8.8.8;8.8.4.4"
# Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
# because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
# externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
# announcement lease and the pihole pod land on different nodes, which is most of the time — and
# Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
# INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
# apparent client's budget, and a normal evening of browsing from a handful of machines would trip
# a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
# off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
# limit becomes meaningful again — and should be restored — if Cilium is ever switched to
# `loadBalancer.mode: dsr`, which preserves the client address.
- name: FTLCONF_dns_rateLimit_count
value: "0"
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
# so the first line is a wildcard for the whole zone and each later line is a specific override.
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
# changes nothing.
#
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
# is the annotation that means "this Service publishes its own hostname at its own address":
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
# resolver address, are the complete set of exceptions today.
#
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
#
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
# while resolution for the zone does not work. Confirmed both ways against the live image.
- name: FTLCONF_misc_dnsmasq_lines
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
ingressRoutes:
- host: pihole.${BASE_DOMAIN}
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
+24 -70
View File
@@ -4,11 +4,21 @@
# auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
# Validates against schemas/servicedefinition.schema.json.
#
# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
# already exists: nothing is provisioned, an existing dataset is mounted. See
# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
# requirement ID is deliberately still not typed here — see that document's "gate" note.)
# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
# conformance test is still a Phase-1 gap).
# The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
# on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
# (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
# that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
# driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
# To onboard the running Plex shell now (compute + config + public exposure), the media volume is
# omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
# storage:
# - name: media
# class: bulk # external mount; size omitted (sized outside the cluster)
# then re-render. See the WS-APPS report for the full bulk-media finding.
name: plex
kind: container
source:
@@ -32,40 +42,11 @@ env:
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
- name: ALLOWED_NETWORKS
value: 10.0.0.0/8,192.168.0.0/16
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
# workload). Plex cannot create libraries over the mounted bulk share while the server is
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
# half of that reasoning survived contact with the upstream source.
#
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
#
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
# the running `:latest` image), NOT assumed:
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
# default 0, which continues startup even when a cont-init script fails.
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
- name: PLEX_CLAIM
valueFrom:
secretKeyRef:
name: plex-claim
key: PLEX_CLAIM
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
# single API POST) against the running server; the claim then persists on the /config PVC and
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -85,37 +66,10 @@ storage:
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
# exchange is then skipped on every later start. (The older note here claimed a
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
# failure, and that is why one is committed.)
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
# downloads/ movies/ photos/ tv/).
#
# WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
# so no subPath could remove a privilege that matters, while three mounts would triple the NFS
# mounts per pod and mean a manifest change every time a top-level library is added.
#
# WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
# indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
# costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
# PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
#
# UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
# world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
# (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
# management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
- name: media
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
mountPath: /media
readOnly: true
size: 9Ti # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
# dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
# something true. The real limit is the ZFS pool.
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
dependencies: [] # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
ingressRoutes:
- host: plex.${BASE_DOMAIN}
+14 -416
View File
@@ -4,18 +4,19 @@
# internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
# (bare web UI, no native OIDC).
#
# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
# This entry used to ship config-only, with a comment promising to attach the downloads volume "once
# OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
# `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
# their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
# of the chain with no media volume at all. Measured on prod 2026-09-12:
# Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/
# Downloads\SavePath=/downloads/ `ls /downloads` -> No such file or directory
# i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
# Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
# recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
# storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
# docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
# replicated media handoff to the arr stack). The `bulk` StorageClass exists
# (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
# it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
# is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
# would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
# conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
# and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
# with a one-line add under storage:
# - name: downloads
# class: bulk # external mount; size omitted (sized outside the cluster)
# then re-render. See the WS-APPS report for the full bulk-downloads finding.
#
# VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
# (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
@@ -31,411 +32,13 @@ name: qbittorrent
kind: container
source:
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
env:
- name: PUID
value: "1000" # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
- name: PGID
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
# readable to the EXISTING non-k3s consumers of the same dataset, and land
# group-owned the same way the arrs' own writes do
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
# tested valid against the then-current temporary password; both would have started failing at
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
# report a failed client, not a failed import).
# * The human operator had no stable login: every restart meant reading the pod log.
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
- name: QBT_WEBUI_USERNAME
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-username
- name: QBT_WEBUI_PASSWORD
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-password
lifecycle:
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
#
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
# the container start rather than a clock, because the failure this closes IS a container start:
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
# restore or rebuild in which the app serves a random password and both arrs are broken. The
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
# In-container, before the service starts, none of that exists.
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
# privilege than the CronJob alternative (R-SEC/least-privilege).
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
# keeps in its own state), and the schema documents this hook for exactly that use.
#
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
# credential — because the WebUI is not running yet. That assertion is the most valuable line
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
# mean a schema version bump plus a renderer change — a blast radius across every service's
# golden — to buy something strictly weaker than what the hook already does.
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
#
# DURABILITY, the whole point — all three paths end with the SAME credential:
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
# hook runs and re-asserts over whatever the snapshot contained.
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
# credential into it before the service is serving. Nothing outside git.
#
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
# password is simply now known, fixed, and reproducible from git.
postStart:
exec:
command:
- /bin/bash
- -c
- |
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
#
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
# CLUSTER — but only there:
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
# however it is run.
set -u
CONF=/config/qBittorrent/qBittorrent.conf
SVC=/run/service/svc-qbittorrent
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
# env var for a lookup to read anyway.
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
# logged "the WebUI rejected the durable
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
PORT=8080
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
# credential from git was asserted and accepted on this start.
log() {
echo "[qbt-webui-cred] $*"
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
}
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
# racing to create one it would then not overwrite.
n=0
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
waited=0
attempt=0
while [ "$attempt" -lt 3 ]; do
attempt=$((attempt + 1))
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
# 10 = rewritten, anything else = could not write.
python3 - "$CONF" <<'PY'
import base64, hashlib, hmac, os, stat, sys
conf = sys.argv[1]
user = os.environ.get("QBT_WEBUI_USERNAME", "")
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
if not user or not pw:
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
raise SystemExit(1)
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
# iterations, 16-byte random salt, 64-byte digest, stored as
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
# Get any one of them wrong and the app simply rejects every login.
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
def matches(raw):
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
v = raw.strip().strip('"')
if v.startswith("@ByteArray(") and v.endswith(")"):
v = v[len("@ByteArray("):-1]
try:
salt_b64, digest_b64 = v.split(":", 1)
salt = base64.b64decode(salt_b64, validate=True)
digest = base64.b64decode(digest_b64, validate=True)
except Exception:
return False
if not salt or not digest:
return False
return hmac.compare_digest(
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
try:
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
except FileNotFoundError:
lines = []
def value_of(key):
for i, line in enumerate(lines):
if line.startswith(key + "="):
return i, line.split("=", 1)[1]
return -1, None
ui, uval = value_of(UKEY)
pi, pval = value_of(PKEY)
if uval == user and pval is not None and matches(pval):
raise SystemExit(0) # converged - leave the file (and its salt) alone
salt = os.urandom(SALT_LEN)
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
if SECTION in out:
at = out.index(SECTION) + 1
else:
if out and out[-1].strip():
out.append("")
out.append(SECTION)
at = len(out)
out[at:at] = desired
try:
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
tmp = conf + ".webui-cred.new"
with open(tmp, "w", encoding="utf-8") as fh:
fh.write("\n".join(out) + "\n")
fh.flush()
os.fsync(fh.fileno())
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
# cost the app the ability to save its own settings, so carry the ownership over.
try:
st = os.stat(conf)
os.chown(tmp, st.st_uid, st.st_gid)
os.chmod(tmp, stat.S_IMODE(st.st_mode))
except FileNotFoundError:
os.chown(tmp, int(os.environ.get("PUID", "1000")),
int(os.environ.get("PGID", "1000")))
os.chmod(tmp, 0o664)
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
except OSError as exc:
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
raise SystemExit(1)
raise SystemExit(10)
PY
rc=$?
if [ "$rc" = 0 ]; then
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
elif [ "$rc" = 10 ]; then
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
# have read the old, credential-less conf and be serving a random temporary password.
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
# Nothing to do when the service has not started yet - it will read the new file.
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
# 2026-09-13 failure spent the entire verification budget before the first login and
# then fired attempts 2 and 3 with no wait left at all.
sv=0
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
sv=$((sv + 1)); sleep 1
done
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "qbittorrent-nox was already running - restarting it to load the credential"
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
sleep 3
fi
else
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
exit 0
fi
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
#
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
# conflating them is what made this hook lie once already (see the PORT comment). An
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
# its remaining login attempts to produce two more identical wrong diagnoses.
api_ok=0
while [ "$waited" -lt 120 ]; do
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
waited=$((waited + 1)); sleep 1
done
if [ "$api_ok" != 1 ]; then
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
log "never submitted. Do not rotate the credential on the strength of this line." >&2
exit 0
fi
body=$(curl -sS -m 10 \
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
if [ "$body" = "Ok." ]; then
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
exit 0
fi
log "WebUI rejected the durable credential (login said '$body')" >&2
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
# make the process RE-READ the file, so restart the supervised service and let the next
# attempt re-verify. Without this the retry loop could never change its own outcome.
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
sleep 3
fi
done
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
# app, and the arrs' own download-client health check surfaces a broken credential.
exit 0
podSecurityContext:
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
# the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
# depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
# `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
# and losing write access to it would lose the WebUI password hash and the session state.)
fsGroup: 3000
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
storage:
- name: config
size: 5Gi
class: longhorn # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
# /downloads, which is precisely where this app is ALREADY configured to write.
#
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
# * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
# LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
# link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
# IS the export root. qBittorrent performs no such cross-directory link: it writes into the
# downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
# from this pod's mount namespace without removing any operation it actually performs.
# * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
# both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
# a file created from this pod under /downloads was hardlinked from the sonarr pod between
# /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
# * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
# workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
# a single subPath mount is not the EXDEV trap, two are.
# * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
# whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
# the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
#
# NOT readOnly — this is the one workload whose whole job is WRITING here.
#
# KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
# `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
# chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
# the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
# non-cluster consumers, which should never be a surprise.
#
# PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
# Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
# ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
# path /downloads/, local path /media/downloads/).
#
# THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
# download client itself — which did not exist in EITHER arr until then (both
# `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
# at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
# reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
# present — same torrent, same client.
#
# BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
# from git alone does NOT restore them and no gate here notices their absence. The alternative —
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
# not chosen because it changes a running download client's save path, which the deferral above
# already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
# mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
# decision, not a technical blocker.
#
# RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
# WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
# start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
# same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
- name: downloads
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
# plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
subPath: downloads # the handoff subtree ONLY: no movies/, no tv/, no photos/
mountPath: /downloads # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
# change to config-PVC app state is needed to make this take effect
size: 9Ti # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
# dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
# state it so `kubectl get pv` reports something true
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
# STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
# OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
# HOST netns, not by this pod: there is no pod-side socket to 2049, so no
# CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
# construction — sonarr and radarr have mounted this same export read-write
# for weeks and neither has any CNP naming the NFS server or port 2049
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
# egress gateway off this traffic (its selector is a podSelector), and
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
ingressRoutes:
- host: qbittorrent.${BASE_DOMAIN}
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
@@ -443,11 +46,6 @@ auth:
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in an authorized group (R-IDP-06)
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=5.0.0 <6.0.0" # stay on the qBittorrent 5.x stable line
+5 -82
View File
@@ -9,23 +9,6 @@ name: radarr
kind: container
source:
image: lscr.io/linuxserver/radarr:5.21.1 # pinned stable v5 tag (R-UPD-01); update.range governs advancement
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
# which cannot write a 1000:3000-owned 0755 tree.
env:
- name: PUID
value: "1000" # owner uid of every directory under /mnt/Storage/plex
- name: PGID
value: "3000" # dominant group there; keeps files this writes readable to the
# EXISTING non-k3s consumers of the same export, which stay live
- name: TZ
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
podSecurityContext:
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
fsGroup: 3000
exposure: internal # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
@@ -33,66 +16,11 @@ storage:
- name: config
size: 5Gi
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
# /media/movies (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
#
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Radarr imports by HARDLINKING or
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
# /mnt/Storage/plex/movies/ the library
# /mnt/Storage/plex/downloads/complete/radarr/ where finished releases land
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
# them as two subPath volumes (subPath: movies at /media/movies, subPath: downloads elsewhere) creates
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Radarr then
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
# requirement, not a shortcut. (The cost is stated plainly: Radarr can also write /media/tv,
# which is sonarr's. Narrowing that would cost hardlinks, which is a worse trade.)
#
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
# download client will already agree on absolute paths — which is the other precondition for
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
#
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
# it belongs in its own change with the operator present.
#
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
#
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
# the image re-chowning /config for itself.
- name: media
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
mountPath: /media # => /media/movies (Root Folder) and /media/downloads (import source)
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
# rounded up, so `kubectl get pv` reports something true.
# NOTE (bulk-media PREREQUISITE): docs/13 row 3 places the media library / downloads volume on the
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Radarr deployable today.
dependencies:
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
port: 8080
@@ -107,11 +35,6 @@ auth:
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in the media group (R-IDP-06)
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=5.0.0 <6.0.0" # stay on the Radarr v5 stable line
+26 -911
View File
File diff suppressed because it is too large. Load diff
+11 -31
View File
@@ -78,34 +78,16 @@ env:
# (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
# JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
# client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
# token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
# SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
#
# PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
# semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
# client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
# redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
# reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
# prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
# 192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
# platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
#
# THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
# ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
# client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
# in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
# var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
# SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
# order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
# fail-loud in the way that matters: if the Secret key is missing the Pod never starts
# (CreateContainerConfigError), rather than silently authenticating against nothing. ---
- name: SEMAPHORE_OIDC_CLIENT_SECRET
# token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
# semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
# keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
# kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
# The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
- name: SEMAPHORE_OIDC_PROVIDERS
valueFrom:
secretKeyRef:
name: semaphore-bootstrap
key: oidc-client-secret
- name: SEMAPHORE_OIDC_PROVIDERS
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
key: oidc-providers-json
# The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
# and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
# is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
@@ -135,12 +117,10 @@ auth:
groups:
- platform-admins # access scoped to an authorized group (R-IDP-06)
# WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
# (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
# client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
# The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
# break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
# No secret is embedded here.
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
# in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
# authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
# never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
update:
channel: stable # R-UPD-01
range: ">=2.19.0 <3.0.0" # stay on the 2.x stable line
+5 -82
View File
@@ -9,23 +9,6 @@ name: sonarr
kind: container
source:
image: lscr.io/linuxserver/sonarr:4.0.15 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
# which cannot write a 1000:3000-owned 0755 tree.
env:
- name: PUID
value: "1000" # owner uid of every directory under /mnt/Storage/plex
- name: PGID
value: "3000" # dominant group there; keeps files this writes readable to the
# EXISTING non-k3s consumers of the same export, which stay live
- name: TZ
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
podSecurityContext:
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
fsGroup: 3000
exposure: internal # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
@@ -33,66 +16,11 @@ storage:
- name: config
size: 5Gi
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
# /config), so without this the Longhorn PVC mounted at /data/config sat
# holding nothing but lost+found while the app wrote its entire
# configuration to the container's EPHEMERAL writable layer — destroyed on
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
# /media/tv (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
#
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Sonarr imports by HARDLINKING or
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
# /mnt/Storage/plex/tv/ the library
# /mnt/Storage/plex/downloads/complete/tv-sonarr/ where finished releases land
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
# them as two subPath volumes (subPath: tv at /media/tv, subPath: downloads elsewhere) creates
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Sonarr then
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
# requirement, not a shortcut. (The cost is stated plainly: Sonarr can also write /media/movies,
# which is radarr's. Narrowing that would cost hardlinks, which is a worse trade.)
#
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
# download client will already agree on absolute paths — which is the other precondition for
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
#
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
# it belongs in its own change with the operator present.
#
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
#
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
# the image re-chowning /config for itself.
- name: media
class: bulk
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
mountPath: /media # => /media/tv (Root Folder) and /media/downloads (import source)
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
# rounded up, so `kubectl get pv` reports something true.
# NOTE (bulk-media PREREQUISITE): docs/13 row 2 places the media library / downloads volume on the
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Sonarr deployable today.
dependencies:
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
port: 8080
@@ -107,11 +35,6 @@ auth:
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
groups:
- media-users # authenticated AND in the media group (R-IDP-06)
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
- platform-admins # operators of these services must not be locked out of them
update:
channel: stable # R-UPD-01
range: ">=4.0.0 <5.0.0" # stay on the Sonarr v4 stable line
+3 -80
View File
@@ -27,31 +27,9 @@ make -C ci test # unit tests for these scripts themselves (ci/tests/*.tes
```
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
selected *by* the requirements a change touches — because **the change most likely to break it
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
exactly that change.
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
and the runner unions every declaring test into the run. Adding the annotation is the only step.
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
full rationale and the incidents behind it.
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
spec-immutability enforcement). See `conformance/tests/README.md`.
## Exit-code contract
@@ -59,47 +37,6 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
the gate stays honest in Phase 0 (no fake green).
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
merge-approved. Fail-open in the one component whose whole job is to fail closed.
| verdict | exit | meaning |
|---|---|---|
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
caller that must distinguish them invokes `./ci/gate.sh` directly.
### Which skips block, and why — a skip is not one thing
| stage | a skip there means | verdict |
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
strictness.
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
### The touched-requirement subset can also be *undeterminable* (exit 3)
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
@@ -150,8 +87,6 @@ ci/
lint.sh Stage: yaml/lint
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
other stage can see — permanently-firing and structurally INERT rules.
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
exits 3 LOUDLY rather than guessing when the diff base is unreliable
@@ -187,15 +122,3 @@ its owner delivers a runnable entrypoint.
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
turns any unverified cluster-needing touched requirement into a hard block.
All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
it, which can only make the gate stricter about what it says it verified.
| switch | default | effect |
|---|---|---|
| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |
+1 -5
View File
@@ -47,11 +47,7 @@ fi
# (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
# invocation with one aggregated report (no per-layer looping needed). Artifacts are then
# COLLECTED from the fixed location below.
# --include-scope: union the touched subset with every L2/L3 test DECLARING
# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
: "${WHOLE_REPO_SCOPE:=whole-repo}"
"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
if [ -f "${CONF_ART}/report.junit.xml" ]; then
+10 -163
View File
@@ -7,75 +7,6 @@
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
# NOT counted as green (HARD RULE 4: don't fake green)
#
# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
# SKIPPED only printed a warning ("— not green") and the gate then printed
# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
# this shape of non-answer — the offline stages just did not use it.
#
# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was
# green, and the R-TEST-03 subset was checked.
# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT
# MUST (see the per-stage rules below), or the
# decision input is unusable.
# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so
# greenness was NOT ESTABLISHED. Never a pass;
# non-zero so no automation can read it as approval.
# `3` is this repo's existing "not run / not
# established" code (ci/lib.sh), now spoken by the
# gate as a whole and not only by its stages.
# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on
# base) — a push-to-main CI run. Stages were green;
# no subset was checked, and the headline says so.
#
# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
# change exists to remove, just relocated.
#
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
#
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
#
# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, so
# the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
# PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
# because on the live runner it is not a defect anyone can fix from inside this repo: the
# workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
# lives in merge-gate.yaml), and blocking would leave the only enforced status check
# permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
# CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
# remedy exists and the message names it: check the spec repo out beside this one, or point
# SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
# Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
# hard block once that checkout is wired.
#
# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
# NOT an accident: it is the environment DECLARING that the capability is absent, which
# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
# to run", so it does not deny a PASS — the touched requirements it would have covered
# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
# it already did. That asymmetry is the whole distinction this file now draws:
# DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
# the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
# verified" when a layer was declared out of scope.
#
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
# available, the gate reports those requirements UNVERIFIED and — unless
@@ -89,7 +20,6 @@ source "${CI_DIR}/lib.sh"
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
: "${REQUIRE_CLUSTER_TESTS:=false}"
: "${REQUIRE_SPEC_VERIFICATION:=false}"
declare -A OUTCOME
run_stage() {
@@ -117,13 +47,6 @@ run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
log "==== gate decision ===="
BLOCK=0
# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
NOT_ESTABLISHED=()
# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
# "PASS" is never mistaken for "everything was verified".
DECLARED_GAPS=()
# --- R-TEST-03 input: the touched-requirement subset --------------------------
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
@@ -141,80 +64,37 @@ rm -f "$_reason_file"
TOUCHED=()
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
UNDECIDED=0
SUBSET_BROKEN=0
if [ "$TOUCHED_RC" -eq 3 ]; then
UNDECIDED=1
echo >&2
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
elif [ "$TOUCHED_RC" -ne 0 ]; then
# Any OTHER non-zero exit is a hard defect in the subset computation itself — an
# incomplete path->requirement map (a changed path matches no row) or an invalid row
# would report exactly this. The reasoning is taken from the unlanded
# fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
# any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
# printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
# `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
# already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
# cannot be computed is NOT an empty subset.
SUBSET_BROKEN=1
echo >&2
err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED[*]}"
fi
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
# file header for why this one is not upgraded to a block by CI detection alone.
# A skip means the spec repo wasn't available to verify — surfaced, not green.
case "${OUTCOME[spec-immutability]}" in
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
skip)
if truthy "$REQUIRE_SPEC_VERIFICATION"; then
err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
BLOCK=1
else
warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
warn " this run cannot be a PASS. Check the spec repo out beside this one (or set"
warn " SPEC_REPO) to verify it."
summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
NOT_ESTABLISHED+=("spec-immutability")
fi ;;
skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
pass) ok "spec-immutability: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
for s in lint l0 render-golden l1; do
case "${OUTCOME[$s]}" in
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
skip)
if strict_skips; then
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
BLOCK=1
else
warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
NOT_ESTABLISHED+=("$s")
fi ;;
skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
pass) ok "${s}: green" ;;
esac
done
# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
# than the stage failing to run where it could have — see the file header.
# L2/L3 conformance: cluster-gated.
case "${OUTCOME[conformance]}" in
fail)
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
@@ -225,22 +105,18 @@ case "${OUTCOME[conformance]}" in
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
else
if [ "${#TOUCHED[@]}" -gt 0 ]; then
reqs=("${TOUCHED[@]}")
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
warn " (no cluster — Phase 0). These are NOT certified green."
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
if truthy "$REQUIRE_CLUSTER_TESTS"; then
if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
BLOCK=1
else
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
fi
else
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
fi
fi ;;
esac
@@ -256,34 +132,13 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
# An incomplete/invalid path map is not an environment problem to skip past: the change under
# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
if [ "$SUBSET_BROKEN" -eq 1 ]; then
err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
echo >&2
# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
# reasons this is not a green run, and the stronger, non-zero one must be the headline.
if [ "$BLOCK" -ne 0 ]; then
err "MERGE GATE: BLOCKED"
summary ""
summary "**Result: BLOCKED** — see failures above."
exit 1
elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
err " Stages that did not run: ${NOT_ESTABLISHED[*]}"
err " Nothing FAILED, and nothing is certified either: a stage that did not run is not"
err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
err " merge-approved. Fix the environment (install the pinned tool / wire the seam /"
err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
err " ./ci/gate.sh || [ \$? -eq 3 ]"
summary ""
summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
exit 3
elif [ "$UNDECIDED" -eq 1 ]; then
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
@@ -293,16 +148,8 @@ elif [ "$UNDECIDED" -eq 1 ]; then
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
exit 0
else
ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
ok " R-TEST-03 touched-requirement subset was checked."
if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
ok " NOT verified here, because the environment DECLARED it out of scope (never a silent"
ok " gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
else
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
fi
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
summary ""
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
exit 0
fi
+44 -43
View File
@@ -10,18 +10,6 @@
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
# a per-resource policy cannot express (secret hygiene across every
# tracked file, least-privilege RBAC, spec-immutability enforcement).
# Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
# test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
# replaces what used to be a hand-written "UNCONDITIONAL" stage per
# whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
# bulk-tier): three near-identical 12-line rationales, three hardcoded
# check.sh paths and three copies of the same rc 0/3/* case block, which
# did not scale and — worse — failed SILENTLY, because whoever added the
# next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
# test declares its own scope and the runner enforces it. The reasoning and
# the incidents those stages recorded were moved INTO the tests they
# describe (see each chainsaw-test.yaml's `scope` annotation) and the
# general mechanism is documented in conformance/lib/common.sh.
#
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
@@ -95,45 +83,58 @@ fi
# binding, the spec-immutability gate's own behaviour) live in the conformance
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
# through the same single-command runner CI and a laptop both use (R-TEST-02).
#
# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
# --------------------------------------------------------------
# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
# whose inputs live under the subtree its requirements map to. It is WRONG for one that
# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
# most likely to break the invariant attributes no requirement at all, so requirement-scoped
# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
# in every L0 test that DECLARES that scope, so those run on every gate invocation.
#
# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
# the gate's own rc 3 below, never to rc 0.
: "${WHOLE_REPO_SCOPE:=whole-repo}"
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
summary "- l0-static: **SKIPPED** (conformance runner missing)"
if [ "$rc" -eq 0 ]; then rc=3; fi
else
load_touched_requirements
l0_static_rc=0
"$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
case "$l0_static_rc" in
0) ok "L0 static conformance checks passed"
summary "- l0-static: **passed**" ;;
3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
ok "L0 static conformance checks passed"
summary "- l0-static: **passed**"
else
err "L0 static conformance checks failed"
summary "- l0-static: **FAILED**"
rc=1
fi
fi
# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
# a legitimate manifest, and that no expansion silently empties today.
#
# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
# attributes no requirement, so the requirement-scoped selection would filter this check
# out of exactly the change that introduces the bug. The invariant is whole-repo (same
# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
if [ ! -f "$SUBST_CHECK" ]; then
skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
summary "- subst-safety: **SKIPPED** (check missing)"
if [ "$rc" -eq 0 ]; then rc=3; fi
else
subst_rc=0
bash "$SUBST_CHECK" || subst_rc=$?
case "$subst_rc" in
0) ok "Flux postBuild substitution safety passed"
summary "- subst-safety: **passed**" ;;
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
summary "- subst-safety: **SKIPPED** (nothing to audit)"
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
*) err "L0 static conformance checks failed"
summary "- l0-static: **FAILED**"
rc=1 ;;
*) err "Flux postBuild substitution safety FAILED"
summary "- subst-safety: **FAILED**"; rc=1 ;;
esac
# Per-check step-summary lines, so the gate output still NAMES what ran — the generated
# replacement for the one hardcoded `summary` line each deleted stage carried.
emit_offline_summary
fi
case "$rc" in
+3 -25
View File
@@ -29,18 +29,7 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
# --include-scope: union the touched subset with every L1 test DECLARING
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
: "${WHOLE_REPO_SCOPE:=whole-repo}"
l1_rc=0
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
${req_args[@]+"${req_args[@]}"} || l1_rc=$?
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -50,16 +39,5 @@ if [ -d "${CONF_ART}/coverage" ]; then
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
fi
case "$l1_rc" in
0) ok "L1 rendering checks passed"
summary "- l1-render: **passed**"
emit_offline_summary ;;
3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
emit_offline_summary
exit 3 ;;
*) err "L1 rendering checks FAILED (rc=${l1_rc})"
summary "- l1-render: **FAILED**"
emit_offline_summary
exit "$l1_rc" ;;
esac
ok "L1 rendering checks passed"
summary "- l1-render: **passed**"
+9 -81
View File
@@ -72,83 +72,20 @@ entrypoint_ready() {
return 1
}
# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
# silently declaring a capability "available", so a typo can only make the gate stricter about
# what it claims to have verified, never looser.
truthy() {
case "${1:-}" in
true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
cluster_available() {
case "${CLUSTER_AVAILABLE:-false}" in
true|1|yes) return 0 ;;
*) return 1 ;;
esac
}
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
#
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
# before the gate runs, and every seam is on `main`. A stage that skips here is a
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
#
# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
#
# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
# env-var sniffing can get it wrong.
# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
# like CI" is exactly the unreliable inference this gate must not contain.
#
# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
# and only for the one declared-environmental skip (see its comment). A workstation misread as
# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
# locally to rehearse CI strictness.
: "${GATE_STRICT_SKIPS:=auto}"
strict_skips() {
case "$GATE_STRICT_SKIPS" in
auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
*) truthy "$GATE_STRICT_SKIPS" ;;
esac
}
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
# --- Offline (L0/L1) per-check step summary -----------------------------------
# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
# gate's output still NAMES every check that ran and its verdict.
#
# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
# naming is generated for EVERY offline check rather than for the three that had a stage.
: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
emit_offline_summary() {
local f="${1:-$OFFLINE_SUMMARY_FILE}" line
[ -f "$f" ] || return 0
while IFS= read -r line; do
[ -n "$line" ] && summary " $line"
done < "$f"
return 0
}
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
# (ready to splice into a conformance/run.sh invocation).
@@ -174,16 +111,7 @@ load_touched_requirements() {
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
elif [ "$rc" -ne 0 ]; then
# Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
# path->requirement map that cannot classify a changed path, once that map fails closed).
# The FULL layer still runs here as a superset — nothing goes untested — but this stage must
# not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
# merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
# (47068ee), whose gate-side half this branch lands.
warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
warn " ci/touched-requirements.sh so every changed path is classified."
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED_REQS[*]}"
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
-214
View File
@@ -1,214 +0,0 @@
#!/usr/bin/env python3
# =============================================================================
# ci/render-keycloak-client-urls.py
#
# Render platform/keycloak/client-urls.yaml FROM platform/keycloak/keycloak-realm-import.yaml.
#
# WHY THIS EXISTS
# ---------------
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
# Done=True / Started=False it never re-runs, so a redirect URI changed in git is
# never applied to the realm. The client-secret half of that failure class is
# already healed by a level-triggered CronJob (platform/keycloak/client-secret-
# reconciler.yaml). This renderer supplies the DESIRED STATE that the same CronJob
# needs to heal the URL half.
#
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
# automountServiceAccountToken: false. It has NO Kubernetes API access at all, by
# design (R-SEC-02) — so it cannot read the KeycloakRealmImport CR to learn what
# the realm is supposed to look like. Its inputs must be MOUNTED. Hence a
# ConfigMap.
#
# A hand-authored ConfigMap would be a SECOND SOURCE OF TRUTH for every public
# hostname in the realm, free to disagree with the realm import — and "two
# declarations that can disagree" is the same shape as the bug this whole effort
# is closing. So the ConfigMap is GENERATED from the realm import and the
# generation is GATED: conformance/tests/l0-static/keycloak-client-secret-
# reconciler/check.sh runs this script with --check on every merge, and the merge
# fails if the committed file is not what this renderer produces. That is the same
# render==golden idiom catalog/render.py + ci/render-golden.sh already use for
# apps/.
#
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. If a URL in the realm import is
# written as a Flux substitution token (https://grafana.${BASE_DOMAIN}/...), the
# token is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files
# are rendered by the SAME Flux Kustomization (./platform-config, whose
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the
# same pass with the same values and cannot disagree about what BASE_DOMAIN is.
# This renderer never parses, splits, or reconstructs a hostname, and never reads
# cluster-vars — so there is no second place for the domain to be decided.
#
# USAGE
# python3 ci/render-keycloak-client-urls.py # print the rendered file
# python3 ci/render-keycloak-client-urls.py --write # write it (regenerate)
# python3 ci/render-keycloak-client-urls.py --check # exit 1 if it has drifted
# =============================================================================
from __future__ import annotations
import argparse
import difflib
import json
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(__file__).resolve().parent.parent
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
OUT_FILE = "platform/keycloak/client-urls.yaml"
DATA_KEY = "client-urls.json"
CONFIGMAP_NAME = "keycloak-client-urls"
NAMESPACE = "keycloak"
# The client fields this renderer exports, and therefore the ONLY fields the
# reconciler may write. Every one of them is an environment-derived URL: it changes
# when BASE_DOMAIN changes, and a stale value breaks OIDC login (a wrong
# redirectUri) or silently widens trust (a stale wildcard left behind).
#
# DELIBERATELY ABSENT: `secret` (healed by the secret pass from the SOPS Secret,
# and it must never appear in a ConfigMap), and every non-URL field —
# protocolMappers, defaultClientScopes, attributes, flags. Those are not
# environment-derived, so nothing re-renders them on a domain flip; keeping them
# out keeps the reconciler's write surface as narrow as the problem.
LIST_FIELDS = ("redirectUris", "webOrigins")
STR_FIELDS = ("rootUrl", "baseUrl", "adminUrl")
def render(root: Path) -> str:
"""Return the full text of client-urls.yaml as it should be committed."""
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
if imp is None:
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
clients = (imp.get("spec") or {}).get("realm", {}).get("clients") or []
if not clients:
# Fail closed. Emitting an empty desired state would make the reconciler
# believe every client is unmanaged and silently stop reconciling URLs.
raise SystemExit("FATAL: %s declares zero clients — refusing to render an "
"empty desired state" % IMPORT_FILE)
want: dict[str, dict] = {}
for c in clients:
cid = c.get("clientId")
if not isinstance(cid, str) or not cid:
raise SystemExit("FATAL: %s has a client with no clientId" % IMPORT_FILE)
entry: dict[str, object] = {}
for f in LIST_FIELDS:
if f not in c:
continue # not declared => NOT MANAGED (never forced empty)
v = c[f]
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
raise SystemExit("FATAL: %s client %r field %s is not a list of strings"
% (IMPORT_FILE, cid, f))
entry[f] = list(v) # verbatim, order as authored
for f in STR_FIELDS:
if f not in c:
continue
v = c[f]
if not isinstance(v, str):
raise SystemExit("FATAL: %s client %r field %s is not a string"
% (IMPORT_FILE, cid, f))
entry[f] = v
if entry:
want[cid] = entry
if not want:
raise SystemExit("FATAL: no client in %s declares any of %s — refusing to render "
"an empty desired state"
% (IMPORT_FILE, ", ".join(LIST_FIELDS + STR_FIELDS)))
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
return HEADER + indented + "\n"
HEADER = """---
# =============================================================================
# GENERATED FILE — DO NOT EDIT BY HAND.
#
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm.clients)
# regenerate: python3 ci/render-keycloak-client-urls.py --write
# gated by: conformance/tests/l0-static/keycloak-client-secret-reconciler
# (runs this renderer with --check on every merge)
#
# WHAT THIS IS: the DESIRED STATE for every environment-derived client URL in the
# `homelab` realm — redirectUris, webOrigins, rootUrl, baseUrl, adminUrl. The
# client-secret reconciler CronJob mounts it and makes the live realm match it.
#
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
# once Done=True, so a redirect URI edited in git never reaches the realm. Prod's
# stored redirect URIs were measured still pointing at the pre-migration hostname
# while git declared the new one, and every OIDC login would have failed with a
# redirect_uri mismatch. The reconciler is the level-triggered loop that closes
# that; this file is what it reconciles TOWARDS.
#
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of every public
# hostname would be a second source of truth, free to disagree with the realm
# import. It is rendered from the import and the render is gated, so the two
# cannot drift. URL strings are copied VERBATIM — if the source uses a
# ${BASE_DOMAIN} token it appears here unchanged, and both files are substituted
# by the SAME Flux Kustomization (./platform-config), so the domain is decided in
# exactly one place.
#
# A field that is NOT declared on a client in the realm import is simply ABSENT
# here, and the reconciler leaves it alone. Absence means UNMANAGED, never
# "force to empty".
# =============================================================================
apiVersion: v1
kind: ConfigMap
metadata:
name: %s
namespace: %s
labels:
app.kubernetes.io/name: %s
app.kubernetes.io/part-of: keycloak
data:
%s: |
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME, DATA_KEY)
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
g = ap.add_mutually_exclusive_group()
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
g.add_argument("--check", action="store_true",
help="exit 1 if the committed file has drifted from the source")
args = ap.parse_args()
want = render(ROOT)
target = ROOT / OUT_FILE
if args.write:
target.write_text(want, encoding="utf-8")
print("wrote %s" % OUT_FILE)
return 0
if args.check:
if not target.is_file():
print("CHECK FAIL: %s does not exist. Run: python3 "
"ci/render-keycloak-client-urls.py --write" % OUT_FILE, file=sys.stderr)
return 1
have = target.read_text(encoding="utf-8")
if have == want:
print("client-urls.yaml is in sync with %s" % IMPORT_FILE)
return 0
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
"realm on a desired state the realm import does not declare.\n"
"Regenerate with: python3 ci/render-keycloak-client-urls.py --write\n"
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
fromfile="committed " + OUT_FILE,
tofile="rendered from " + IMPORT_FILE):
sys.stderr.write(line)
return 1
sys.stdout.write(want)
return 0
if __name__ == "__main__":
raise SystemExit(main())
-189
View File
@@ -1,189 +0,0 @@
#!/usr/bin/env bash
# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
#
# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
# none covered the DECISION.
#
# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
# no network, in milliseconds.
#
# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
#
# Run: ci/tests/gate-decision.test.sh (or `make -C ci test`)
set -uo pipefail
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
CI="${TMP}/ci"; mkdir -p "$CI"
REPO="${TMP}/repo"; mkdir -p "$REPO"
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
for stage in spec-immutability lint l0 render-golden l1 conformance; do
cat > "${CI}/${stage}.sh" <<'STUB'
#!/usr/bin/env bash
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
exit "${!v:-0}"
STUB
chmod +x "${CI}/${stage}.sh"
done
# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
# real script does), TR_OUT (the touched requirement list on stdout).
cat > "${CI}/touched-requirements.sh" <<'STUB'
#!/usr/bin/env bash
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
exit "${TR_RC:-0}"
STUB
chmod +x "${CI}/touched-requirements.sh"
# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
# whether the machine running it happens to look like CI.
OUT="" RC=0
run_gate() {
local summary_file="${TMP}/summary.md"; : > "$summary_file"
OUT="$(env -i PATH="$PATH" HOME="$HOME" \
REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
"$@" bash "${CI}/gate.sh" 2>&1)"
RC=$?
SUMMARY="$(cat "$summary_file")"
}
# assert_verdict <headline> <expected-rc> <label>
assert_verdict() {
local want="$1" want_rc="$2" label="$3"
if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
elif [ "$RC" -ne "$want_rc" ]; then
bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
else
ok "${label} -> ${want}, exit ${want_rc}"
fi
}
# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
assert_not_pass() {
local label="$1"
if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
}
echo "== 1. the happy path: every stage green"
run_gate
assert_verdict PASS 0 "all stages green + a touched subset"
grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
&& ok "PASS summary states nothing was declared out of scope" \
|| bad "PASS summary does not state the absence of declared gaps"
echo
echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
for s in lint l0 render-golden l1 conformance spec-immutability; do
run_gate "STAGE_RC_${s//-/_}=1"
assert_verdict BLOCKED 1 "${s} FAILED"
done
echo
echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
for s in lint l0 render-golden l1; do
run_gate "STAGE_RC_${s//-/_}=3"
assert_not_pass "${s} SKIPPED (local)"
assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
|| bad "${s}: step summary does not name the skipped stage"
done
echo
echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
for s in lint l0 render-golden l1; do
run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
done
# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
# tool or an unwired seam. Only the declared-environmental skip below can produce it.
run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
echo
echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
run_gate STAGE_RC_spec_immutability=3
assert_not_pass "spec-immutability SKIPPED"
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
echo
echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
run_gate STAGE_RC_conformance=3
assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
grep -qF "conformance-l2-l3" <<<"$OUT" \
&& ok "the PASS headline names the declared L2/L3 gap" \
|| bad "the PASS headline hides the declared L2/L3 gap"
grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
&& ok "the step summary names the declared L2/L3 gap" \
|| bad "the step summary hides the declared L2/L3 gap"
run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
# A misspelled switch must leave the guard OFF (truthy()), never half-on.
run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
echo
echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
assert_not_pass "on-base AND l1 SKIPPED"
assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
echo
echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
for rc in 1 2 4; do
run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
done
run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
|| bad "the step summary does not report an unusable subset"
echo
echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
echo
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
@@ -1,164 +0,0 @@
#!/usr/bin/env bash
# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
#
# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
# requirement subset. Weakening (or deleting) a test is the one change that most needs the
# requirement it certifies to be re-verified, and it was the change least likely to select it.
# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
#
# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
# whole-repo-scoped get selected by the paths that can break them.
#
# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
#
# Run: ci/tests/touched-requirements-test-attribution.test.sh (or `make -C ci test`)
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
SCRIPT="${CI_DIR}/touched-requirements.sh"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
assert_present() {
if grep -qxF "$1" <<<"$2"; then ok "$3"
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
}
assert_absent() {
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
else ok "$3"; fi
}
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
git -C "$TMP" init -q -b main
git -C "$TMP" config user.email [email protected]
git -C "$TMP" config user.name t
git -C "$TMP" config commit.gpgsign false
# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
# walk-up from an arbitrary changed file to the enclosing test dir.
mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
"$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
"$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alpha
labels:
conformance.home.k3s/R-ALPHA-01: "true"
conformance.home.k3s/R-ALPHA-02: "true"
annotations:
conformance.home.k3s/layer: "L0"
spec: {}
YAML
# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
# body-only edit came from rule 3 and not from the content scan.
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
#!/usr/bin/env bash
echo "assert something"
SH
cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: beta
labels:
conformance.home.k3s/R-BETA-07: "true"
spec: {}
YAML
echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
echo "harness helper" > "$TMP/conformance/lib/helper.sh"
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
# ========================================================================================
echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
git -C "$TMP" checkout -q -b body-edit
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
#!/usr/bin/env bash
echo "assert something weaker"
SH
git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
assert_present "R-TEST-03" "$OUT" "the conformance/* path map still applies (R-TEST-03)"
assert_absent "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
assert_absent "R-BETA-07" "$OUT" "an UNRELATED test's labels are NOT attributed"
# ========================================================================================
echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
git -C "$TMP" checkout -q -b nested-edit main
echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
assert_absent "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
# ========================================================================================
echo "== case 3: DELETING a test still attributes what it used to certify"
# Removing the only verifier of a requirement must not be the cheapest way to make the gate
# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
git -C "$TMP" checkout -q -b delete-test main
git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
git -C "$TMP" commit -qm "delete the alpha test"
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
# ========================================================================================
echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
git -C "$TMP" checkout -q -b lib-edit main
echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-TEST-02" "$OUT" "a conformance/lib edit still attributes the harness requirements"
assert_absent "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
assert_absent "R-BETA-07" "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
# ========================================================================================
echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
git -C "$TMP" checkout -q -b infra-edit main
printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
git -C "$TMP" checkout -q -b ci-edit main
printf 'echo hello\n' > "$TMP/ci/thing.sh"
git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set:\n%s\n---\n' "$OUT"
assert_present "R-TEST-06" "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
echo
if [ "$FAIL" -eq 0 ]; then
echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
fi
echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
exit 1
-135
View File
@@ -93,141 +93,6 @@ printf 'touched set (removal):\n%s\n---\n' "$OUT2"
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
# ---------------------------------------------------------------------------------------
# PATH MAP — bootstrap/* (added 2026-09-13)
#
# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
# guard by content rather than by path, so it activates on exactly that change. Removing
# the entry would let the change introducing a guard be gated by nothing, which is the
# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
# silent loss of selection.
#
# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
# map precisely so the other cases isolate the scan, and here the file content cites no
# R-ID at all, so any R-ID in the output came from the PATH.
git -C "$TMP" checkout -q -b feature3 main
mkdir -p "$TMP/bootstrap/flux"
cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
#!/usr/bin/env bash
# a bootstrap-path change that cites no requirement id of its own
echo "flux bootstrap"
SH
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
assert_present "R-STATE-03" "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
assert_present "R-GITOPS-01" "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
# And the neighbouring entry the same test leans on for its other half: clusters/* must
# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
# authored-values sweep over clusters/<env> effective cluster-vars).
git -C "$TMP" checkout -q -b feature4 main
mkdir -p "$TMP/clusters/base"
printf 'kind: ConfigMap\nmetadata:\n name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
# ---------------------------------------------------------------------------------------
# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
#
# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
#
# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
#
# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
# from the PATH and not from the content scan.
# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
# under test here, not just the presence of both arms.
git -C "$TMP" checkout -q -b feature-apps-vms main
mkdir -p "$TMP/apps/vms/demo"
printf 'kind: VirtualMachine\nmetadata:\n name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
assert_present "R-PLAT-03" "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
assert_absent "R-PLAT-02" "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
# A plain ServiceDefinition under apps/ gets the container-workload set.
git -C "$TMP" checkout -q -b feature-apps main
mkdir -p "$TMP/apps/wave-1"
printf 'kind: ServiceDefinition\nmetadata:\n name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
assert_present "R-PLAT-02" "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
assert_present "R-NET-01" "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
assert_present "R-NET-06" "$OUT_APPS" "apps/* attributes the egress-class requirements"
assert_present "R-IDP-02" "$OUT_APPS" "apps/* attributes the auth-mode requirements"
# homepage / headlamp ARE the command-centre surface.
git -C "$TMP" checkout -q -b feature-c2 main
mkdir -p "$TMP/platform/headlamp"
printf 'kind: Deployment\nmetadata:\n name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
git -C "$TMP" checkout -q -b feature-assembly main
mkdir -p "$TMP/platform"
printf 'resources:\n - cilium\n - longhorn\n' > "$TMP/platform/kustomization.yaml"
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
# ---------------------------------------------------------------------------------------
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
#
# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
#
# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
# somehow matched only the digit form would still fail here.
git -C "$TMP" checkout -q -b feature5 main
cat > "$TMP/notes/c2.yaml" <<'YAML'
kind: Test
metadata:
labels:
conformance.home.k3s/R-C2-01: "true"
conformance.home.k3s/R-C2-02: "true"
conformance.home.k3s/R-OBS-04: "true"
YAML
git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
assert_present "R-C2-01" "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
assert_present "R-C2-02" "$OUT5" "the second such ID is attributed too (R-C2-02)"
assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
# noise (over-broad selection, never a false green, but noisy and misleading).
assert_absent "R-1-01" "$OUT5" "a purely numeric workstream segment is not invented from thin air"
echo
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+11 -188
View File
@@ -21,12 +21,8 @@
# the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
# Untouched lines and prose are NOT scanned; see changed_diff() for why.
# 2. A coarse path->requirement fallback for changes that don't self-declare IDs.
# 3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
# labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
# The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
# to the tests themselves: editing or weakening a test must be gated by the requirements
# that test CERTIFIES, not merely by R-TEST-0x.
# untagged changes so nothing silently escapes the gate.
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
@@ -177,23 +173,6 @@ path_requirements() {
case "$1" in
catalog/*) echo "R-GITOPS-02" ;;
schemas/*) echo "R-GITOPS-02" ;;
# apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
# manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
# adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
# checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
# class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
# violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
# catch it at apply time" is post-merge, and the gate exists to catch it before.
#
# ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
# apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
# subject to default-deny (R-NET-01).
apps/vms/*) echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
# Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
# workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
# (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
# default-deny (R-NET-01).
apps/*) echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
platform/kyverno/*) echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
platform/cilium/*) echo "R-NET-01 R-NET-02 R-NET-05" ;;
platform/longhorn/*) echo "R-STATE-01 R-STATE-02" ;;
@@ -201,161 +180,23 @@ path_requirements() {
platform/externaldns/*) echo "R-DNS-02" ;;
platform/velero/*) echo "R-STATE-03" ;;
platform/monitoring/*) echo "R-OBS-01 R-OBS-02" ;;
# platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
# nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
# there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
# for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
# requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
platform/cnpg/*) echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
# vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
# R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
# steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
# annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
# happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
# ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
# when a diff does carry the tags.
platform/vpn-egress/*) echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
# Keycloak is the whole SSO surface: the realm import (declarative identity config),
# the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
# the client-secret incident, in which editing platform/keycloak/ attributed NO
# requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
# subset — the change that could break every OIDC login was gated by nothing.
# R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
# its off-cluster backup credential, not only the realm.
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
# The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
# rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
# platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
# itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
# docs/12-command-center-ui.md), so editing them must select the C2 tests.
platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
# KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
platform/kubevirt/*) echo "R-PLAT-03 R-OBS-01" ;;
# The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
# TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
platform/gateway/*) echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
# CoreDNS is cluster DNS resolution (R-DNS-01).
platform/coredns/*) echo "R-DNS-01" ;;
# crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
# its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
platform/crafty/*) echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
# Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
# runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
# resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
# is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
# runbook change is an observability change.
runbooks/*) echo "R-OBS-04" ;;
# platform-config/ is the env-specific configuration Flux substitutes into and applies.
platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
# platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
# Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
# file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
# change. That is the most consequential single-line edit in the repo and it attributed nothing.
platform/kustomization.yaml) echo "R-GITOPS-01" ;;
# .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
# here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
.sops.yaml) echo "R-SEC-01 R-GITOPS-01" ;;
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
# infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
# the nodes are provisioned as code was filtered out of every change that could break it.
# That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
# chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
infra/*) echo "R-ENV-01" ;;
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
# bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
# was written against a file that is not here, and that is corrected on 2026-09-13:
# it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
# `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
# BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
# on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
# gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
#
# WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
# its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
# path, so the moment a standup guard lands under bootstrap/ that test's coupling half
# activates — and this entry is what selects the test for the change that lands it.
# Deleting the entry would mean the change introducing a placeholder guard is gated by
# nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
# value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
# descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
# Locked down by ci/tests/touched-requirements.test.sh's path-map section.
# bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
# bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
# enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any
# already-mapped path scoped the L0 run to that other path's requirements and silently
# excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists
# precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that
# stopped matching when the value it guards changed, in a different file) is exactly the
# class that omission would hide.
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
# R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
# was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
# Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
# carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
# surface is ci/ — so a ci/ change must attribute it, or the test that proves the
# spec-immutability mechanism is still load-bearing runs on nothing that can break it.
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
*) : ;;
esac
}
# ---------------------------------------------------------------------------------
# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
#
# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
# Weakening a test is the one change that most needs the requirement it certifies to be
# re-verified, and it was the change least likely to select it.
#
# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
#
# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
# this function can only ADD requirements, never remove one.
: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}" # == CONF_LABEL_PREFIX, conformance/lib/common.sh
file_text_head_or_base() {
local path="$1"
git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
return 0
}
# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
# Walks UP from the changed file to the nearest ancestor directory carrying a
# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
test_requirements_for_path() {
local f="$1" dir yaml text
case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
dir="${f%/*}"
while :; do
case "$dir" in
conformance/tests|conformance|.|"") return 0 ;;
*/*) ;;
*) return 0 ;;
esac
yaml="${dir}/chainsaw-test.yaml"
text="$(file_text_head_or_base "$yaml")"
if [ -n "$text" ]; then
# Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
# content scan applies (changed_diff() rule 3): a label shown inside a header example is
# documentation. Only a real YAML mapping line counts.
printf '%s\n' "$text" \
| grep -vE '^[[:space:]]*#' \
| sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
| tr '\n' ' '
return 0
fi
dir="${dir%/*}"
done
}
main() {
local files reqs="" rc=0 base_sha head_sha local_main merge_base
@@ -389,27 +230,9 @@ main() {
while IFS= read -r f; do
[ -n "$f" ] || continue
reqs+=" $(path_requirements "$f")"
# Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
# requirement labels, so weakening a test is gated by what the test certifies.
reqs+=" $(test_requirements_for_path "$f")"
done <<< "$files"
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
#
# THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
# contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
# in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
# to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
# Every other consumer in the repo already allows the digit: test_requirements_for_path()
# twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
# conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
# dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
# retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
# path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
# filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
# reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
# platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
# (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
# `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
@@ -423,7 +246,7 @@ main() {
| grep -vE '^(\+\+\+|---)' \
| sed -E 's/^[+-]//' \
| grep -vE '^[[:space:]]*#' \
| grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
| grep -hoE 'R-[A-Z]+-[0-9]+' \
| tr '\n' ' ' || true)"
# Normalize: split, unique, sort. Emit to stdout.
-9
View File
@@ -21,15 +21,6 @@ YAMLLINT_VERSION="1.35.1"
KUBECTL_VERSION="1.36.4"
YQ_VERSION="4.44.6"
# Alert-rule unit tests (promtool test rules) — the L0 check
# conformance/tests/l0-static/alert-rules-unit-tested runs every
# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
# Shipped inside the prometheus release tarball, not as a separate release.
PROMTOOL_VERSION="3.14.0"
# ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
# pinned here so CI and a laptop validate against the exact same validator.
CHECK_JSONSCHEMA_VERSION="0.29.4"
+3 -6
View File
@@ -34,9 +34,6 @@ spec:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
# and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
# overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
# clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
# velero Deployment actually mounts. See clusters/dev/kustomization.yaml.
# SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
# environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.
+49 -164
View File
@@ -96,24 +96,9 @@ data:
#
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
# UNSUBSTITUTED path would ship the literal string into the cluster.
#
# ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
# UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
# live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
# `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
# tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
# the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
# `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
# ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
# the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
# ship the literal string into the cluster.
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
#
@@ -161,70 +146,30 @@ data:
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
# PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
# THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
# terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
# LAN clients are pointed at this address as their resolver, and it has to stay put across
# reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
#
# WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
# the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
# answers where services live" with "a service", and it would put LAN resolution behind the
# Gateway Service's own lifecycle. Keep them separate.
#
# dev -> 192.168.1.56 (base default; next free after crafty .55 in dev's .53-.60 pool)
# prod -> 192.168.2.23 (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
PIHOLE_DNS_VIP: "192.168.1.56" # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
# The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
# peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
# from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
# CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
# same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG
# Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's
# backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default).
#
# dev -> 192.168.1.0/24 (Proxmox LAN; nodes .50/.51/.52)
# prod -> 192.168.2.0/24 (prod LAN; nodes .10/.11/.12)
LAN_CIDR: "192.168.1.0/24" # ingress scope for the PiHole DNS listener
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
# library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
# (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
# `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
# declared in the ServiceDefinition, because the path is identical in every environment and only
# the ADDRESS is not.
# OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time
# being until a better on-prem backup solution is created". Two reasons beyond cost:
# 1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside
# pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change.
# 2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a
# physical failure domain with dev. One host loss took the cluster AND its backups. R2 is
# genuinely off-site.
# Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) +
# 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS
# stay free and R-STATE-03 has no cost disincentive to stay proven.
#
# WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
# The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
# dev nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
# NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
# /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
# per the convention of the LAN block above.
# prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
# 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
# the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
# DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
# So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
# blast radius of shipping it unset, which is NOT the usual "fails to apply".
BULK_NFS_SERVER: "192.168.1.9"
# The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it
# means pointing this knob back at http://192.168.1.9:9000 temporarily.
S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
# and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
# network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
# `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
# S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
# exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
# string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
# accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
# is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
# against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
# value. Prod overrides to `auto`.
S3_BACKUP_REGION: "us-east-1"
# S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2.
# This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for
# MinIO and wrong for R2, which is why it is a knob now.
S3_BACKUP_REGION: "auto"
# --- storage (ADR-0003) ---
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
@@ -278,98 +223,38 @@ data:
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
VELERO_RETENTION: "168h" # dev short | prod long
# ==========================================================================================
# OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
# writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
# ==========================================================================================
# Nothing about the object layout was per-environment before this: Velero's bucket was the
# hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
# `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
# have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
# that layout and:
# (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
# (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
# mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
# dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
# BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each
# other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev.
#
# WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
# the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
# barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
# CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
# hold the same bucket name a second time would be a rotation waiting to drift — the exact class
# of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
# no two of them can contradict each other.
# Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before:
#
# BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
# `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
# a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
# the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
# /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
# from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
# BSL is unchanged by this parameterization rather than merely equivalent to it.
# VELERO — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and
# `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into
# itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would
# therefore adopt dev's backups and delete them on schedule.
# CNPG — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the
# CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`,
# so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers
# interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
# dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
#
# PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
# Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is
# worth explaining, because the earlier reasoning was the opposite:
#
# s3://k3s-prod-backup/velero/… Velero (kopia repo + backup metadata)
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG WAL + base backups
# While the target was the MinIO, dev's base value HAD to be the bucket root — its existing
# backups lived there and giving dev a prefix would have ORPHANED the very backups its proven
# restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`).
#
# The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
# prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
# renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
# real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
# bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
VELERO_BACKUP_BUCKET: "velero" # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
VELERO_BACKUP_PREFIX: "" # dev: bucket root (live layout) | prod: "velero"
CNPG_BACKUP_PATH: "s3://cnpg" # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
# ------------------------------------------------------------------------------------------------
# WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
# ------------------------------------------------------------------------------------------------
# The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
# it exists because the endpoint moved per-environment while the credential did not.
# Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no
# reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root
# free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently
# colliding with a real environment's backups.
#
# THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
# <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
# by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
# differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
# credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
# signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
# because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
# so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
# spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
# either environment's key silently breaks the other; that is the structural reason for the split,
# independent of dev's eventual decommission.
#
# THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
# base resources; nothing here forks a manifest per environment). The consumer names one:
#
# velero: credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
# CNPG: s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
#
# ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
# quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
# rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
# UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
# or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
# names are spelled out in full in each kustomization.yaml beside the resources.
#
# THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
# which are the objects that already exist and that the live dev cluster is running. A default
# naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
# Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
# CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
# again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
# Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
#
# An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
# not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
# gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
# "-prod" — so neither inherits it silently.
#
# ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
# `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
# `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
BACKUP_CREDENTIALS_SUFFIX: "" # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
# *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while
# the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are
# one change; do not split the commit.
VELERO_BACKUP_PREFIX: "dev" # dev: "dev" | prod: "prod"
CNPG_BACKUP_PATH: "s3://cnpg/dev" # dev: s3://cnpg/dev | prod: s3://cnpg/prod
# --- placement ---
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
+2 -6
View File
@@ -34,12 +34,8 @@ spec:
kind: GitRepository
name: flux-system
path: ./platform/homepage
# LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
# references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
# configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
# the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
# front door points at a host that does not exist — while Flux still reports this Kustomization
# Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
# Homepage references no ${VARS} today; the substitution is retained for parity with the other
# tiers / future overlay knobs.
postBuild:
substituteFrom:
- kind: ConfigMap
-7
View File
@@ -50,13 +50,6 @@ resources:
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
# Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
# runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
# this tier existed that path was a 404 and every alert handed the operator a dead link. Same
# non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
# purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
# see clusters/base/runbooks.yaml.
- runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config)
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
-57
View File
@@ -1,57 +0,0 @@
---
# Flux Kustomization: alert runbooks — reconciles ./runbooks.
#
# WHAT THIS SERVES. Every alert rule in this repo carries
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
# human handed them a dead link at the moment they needed help. This Kustomization renders the
# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
#
# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
#
# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
# IMPORTANT LINE IN THIS FILE.
# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
# the gate as well.
# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
# inheriting the `https-apex` listener's own hostname instead. So this tier is
# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
# without substitution rather than because of it.
# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
# Kustomization. Split the serving manifests into their own substituted tier and leave the content
# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
#
# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: runbooks
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config # wildcard TLS + Gateway (incl. the https-apex listener) first
sourceRef:
kind: GitRepository
name: flux-system
path: ./runbooks
+14 -81
View File
@@ -106,11 +106,8 @@ patches:
secretRef:
name: sops-age
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
# It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
# MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
# stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
# BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
# out-of-band (name only).
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -122,23 +119,6 @@ patches:
provider: sops
secretRef:
name: sops-age
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
# as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-vms
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
@@ -242,67 +222,20 @@ patches:
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.1.55"
# Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
# (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
# Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
# Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
# end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
# change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
- op: replace
path: /data/BULK_NFS_SERVER
value: "192.168.1.9"
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
# barmanObjectStore and Velero's backupStorageLocation write to.
# OFF-CLUSTER BACKUP TARGET — no op here any more, deliberately.
#
# DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
# Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
# operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
# a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
# would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
# (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
- op: replace
path: /data/S3_BACKUP_ENDPOINT
value: "http://192.168.1.9:9000"
# DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
# reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
# defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
# its proven restore depends on — without any manifest looking wrong.
# This used to pin the on-Proxmox MinIO (http://192.168.1.9:9000). As of 2026-09-05 the operator
# moved backups to Cloudflare R2 "for the time being until a better on-prem backup solution is
# created", and R2 is a single environment-INDEPENDENT endpoint: both clusters write to the same
# buckets and are isolated by VELERO_BACKUP_PREFIX / CNPG_BACKUP_PATH (dev vs prod), not by URL.
# Under R-ENV-02 a value that does not differ per environment belongs in base and must NOT be
# restated here — two copies of one endpoint is just a rotation waiting to drift.
#
# `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
# ignores the value, but changing it is a needless risk against a live, working backup path.
# An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
# are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
# against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
- op: replace
path: /data/S3_BACKUP_REGION
value: "us-east-1"
- op: replace
path: /data/VELERO_BACKUP_BUCKET
value: "velero"
- op: replace
path: /data/VELERO_BACKUP_PREFIX
value: ""
- op: replace
path: /data/CNPG_BACKUP_PATH
value: "s3://cnpg"
# WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
# (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
# its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
# renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
#
# RESTATED EXPLICITLY even though it equals the base default, for the same reason as
# EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
# base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
# signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
# turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
# restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
#
# DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
# 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
# backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
- op: replace
path: /data/BACKUP_CREDENTIALS_SUFFIX
value: ""
# *** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts EMPTY, so Velero writes a
# fresh kopia repository and CNPG takes new base backups; the 30-day Barman window rebuilds from
# zero. The existing MinIO copy is NOT deleted and stays restorable — point S3_BACKUP_ENDPOINT
# back at http://192.168.1.9:9000 (and VELERO_BACKUP_PREFIX back to "") to reach it. Do not merge
# this while dev's proven restore is the only recovery path you are relying on.
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
+40 -247
View File
@@ -113,23 +113,6 @@ patches:
provider: sops
secretRef:
name: sops-age
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
# as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-vms
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
#
# ############################################################################################
@@ -202,56 +185,9 @@ patches:
name: flux-system
namespace: flux-system
spec:
# CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
# from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
# (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
# operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
# laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
#
# THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
# renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
# GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
# too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
# its `main` to the exact commit prod was already running so the source swap changed nothing
# else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
# the authored value agrees. Dev hit the identical trap on 2026-09-05
# (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
#
# Verified before the switch: a throwaway pod INSIDE prod resolved this host through
# in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
# supplied. The workstation being able to reach it proves nothing about pod egress.
# MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
# This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
# carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
# - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
# - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
# ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
# Leaving this on the old name would therefore have severed prod from its OWN GitOps
# source moments after the flip reconciled — self-reference, the same class of trap as
# patching this url live and having the root Kustomization revert it. The fix has to land
# THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
# after it.
# FOLLOW-UP worth doing: point this at the in-cluster Service
# (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
# on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
# IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a
# deadlock the apex flip walked straight into, and it is worth stating precisely.
#
# The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the
# name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have
# served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source
# is broken. Source down -> routes never rename -> source stays down. Observed live:
# `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev".
#
# Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on
# public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future
# domain change cannot sever prod from its own source again.
#
# Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP,
# TLS terminates at the Gateway for external clients only, the GitRepository has no
# secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted
# at rest regardless of transport.
url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
# TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
# above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
ref:
branch: main
- target:
@@ -289,30 +225,20 @@ patches:
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
#
# THE APEX INVERSION IS DELIBERATELY NOT ON THIS BRANCH. Moving prod to the bare apex (and dev
# to dev.home.) is a LIVE MIGRATION of the dev cluster, not a variable change — see
# infra/PROD-PROVISIONING.md §10a. It is kept as a single commit on branch
# `migration/dns-apex-inversion` (9137bbd) and is applied as step D3 of that runbook, with the
# operator present. Everything else on this branch is independent of it, which is the whole
# reason it was split out.
#
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
# trusts a different Keycloak than the one prod actually runs.
- op: replace
path: /data/BASE_DOMAIN
# APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
# decommissioned and released the name. Prerequisites completed and VERIFIED before this
# flip, in this order:
# 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
# 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
# external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
# records are already up to date", so it would NEVER have cleared them. They carried
# `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
# names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
# 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
# 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
# It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
# limit of 5 per identical name set per week.
# NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
# at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
# SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
# lives on `*.keithjackson.dev` and is untouched by any of this.
value: "home.keithjackson.dev"
value: "prod.home.keithjackson.dev"
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
@@ -366,16 +292,8 @@ patches:
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
# size cap before 30d if series volume is high, keeping the disk from filling either way.
- op: replace
# LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
# what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
# not the head's in-flight segments, and not compaction scratch, and a large compaction can need
# roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
# /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
# absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
# at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
# grow the PVC rather than the fraction of it.
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
value: "36GiB"
value: "45GiB"
- op: replace
path: /data/MONITORING_LOKI_PVC_SIZE
value: "50Gi"
@@ -465,163 +383,38 @@ patches:
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.2.22"
# PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
# 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
# 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
# IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
# (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
# broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
# it either, and it is inside the pool block the operator already asserted sits outside the
# SOHO router's DHCP scope.
- op: replace
path: /data/PIHOLE_DNS_VIP
value: "192.168.2.23"
# The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
# NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
# It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
- op: replace
path: /data/LAN_CIDR
value: "192.168.2.0/24"
# ------------------------------------------------------------------------------------------
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05).
#
# THERE IS DELIBERATELY NO `S3_BACKUP_ENDPOINT` OP HERE, and that is a reversal of the rule
# stated at the top of this overlay ("an OMITTED op means prod silently inherits a dev
# address"). That rule exists because base defaults are dev's LAN addresses. R2 is not one: it
# is a single shared, environment-independent endpoint, so under R-ENV-02 it belongs in base and
# overriding it with an identical value would only create two places to change on a rotation —
# the exact drift this repo has been bitten by before. Isolation does NOT come from the
# endpoint; it comes from the two path knobs below.
#
# WHY R2 AND NOT THE MinIO the operator first named: prod is on the SOHO network
# 192.168.2.0/24, which is pfSense's WAN, while the MinIO sits inside pfSense's LAN
# 192.168.1.0/24. Measured from a pod INSIDE prod (the workstation proves nothing about pod
# egress): `curl http://192.168.1.9:9000/minio/health/live` → connection timed out after 15s.
# Reaching it would need a WAN→LAN rule that does not exist. R2 needs no firewall change, and
# unlike that MinIO — which runs on the same Proxmox host as dev's VMs — it is genuinely
# off-site. Verify prod can reach the new endpoint before resuming velero:
# bootstrap/standup.sh ... --s3-endpoint <the R2 endpoint> s3check
#
# THE PATH KNOBS ARE WHAT MAKE ONE SHARED BUCKET SAFE. Nothing about the paths was
# per-environment before 2026-09-05: Velero used one unprefixed bucket, and every CNPG Cluster
# used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both environments have
# keycloak-db / nextcloud-db / semaphore-db. Without these knobs prod would (a) adopt and
# TTL-delete dev's Velero backups and (b) interleave two Postgres servers' WAL into ONE Barman
# stream, corrupting the chain and destroying PITR for both. See clusters/base/cluster-vars.yaml.
# ------------------------------------------------------------------------------------------
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
#
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
# outcomes give DIFFERENT values here:
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
# ever became reachable — mounts something unintended. So it stays a placeholder.
#
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
# on ephemeral numbers.
#
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
#
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
# where these manifests are meant to be proven first.
- op: replace
path: /data/BULK_NFS_SERVER
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
# keeps the base value 192.168.1.9.
#
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
# carries no_root_squash, so a root write would have passed and proved nothing, while
# LSIO images default to PUID 911 and would have gone Ready then failed every import
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
value: "192.168.2.102"
# ==========================================================================================
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
# ==========================================================================================
# Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
# option for prod, for two independent reasons:
# 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
# MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
# nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
# timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
# 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
# a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
# restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
#
# THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
# `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
# exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
# 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
# ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
# 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
# cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
# forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
#
# ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
# address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
# (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
# It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
# there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
# restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
- op: replace
path: /data/S3_BACKUP_ENDPOINT
value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
# R2 requires the region string to be exactly `auto`. (`head-bucket` answers
# `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
# here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
# has always been signed with.
- op: replace
path: /data/S3_BACKUP_REGION
value: "auto"
# ------------------------------------------------------------------------------------------
# OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
# barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
# They are separated by disjoint TOP-LEVEL PREFIXES instead:
#
# s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG: WAL + base backups
#
# The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
# obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
# deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
# per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
# matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
# Dev cannot be reached from any of these: different endpoint AND different bucket names.
# ------------------------------------------------------------------------------------------
- op: replace
path: /data/VELERO_BACKUP_BUCKET
value: "k3s-prod-backup"
- op: replace
path: /data/VELERO_BACKUP_PREFIX
value: "velero"
value: "prod"
# Barman objects land at s3://cnpg/prod/<clusterName>/; dev's at s3://cnpg/dev/<clusterName>/.
- op: replace
path: /data/CNPG_BACKUP_PATH
value: "s3://k3s-prod-backup/barman"
# ------------------------------------------------------------------------------------------
# WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
# ------------------------------------------------------------------------------------------
# The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
# not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
# manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
# gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
# R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
#
# So each credential is now two objects, and this suffix selects prod's:
# velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape
# <db>-backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
# for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
# base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
# than four secret-name knobs deliberately — four could be updated three-quarters of the way,
# leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
# substitution cannot catch a defined-but-stale variable.
#
# THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
# k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
# touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
# the structural reason for the split, independent of dev's eventual decommission.
- op: replace
path: /data/BACKUP_CREDENTIALS_SUFFIX
value: "-prod"
value: "s3://cnpg/prod"
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
-26
View File
@@ -23,9 +23,6 @@ behavioral / L3 fault-injection** (this harness).
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
./run.sh --require-all # phase-exit gate: every requirement must be covered
./run.sh --record # run AND capture the evidence (see "Proof of pass")
./run.sh --layer l0 --include-scope whole-repo # union: the tagged subset PLUS every test
# declaring `conformance.home.k3s/scope: whole-repo`
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
```
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -35,29 +32,6 @@ behavioral / L3 fault-injection** (this harness).
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
those touched requirements are covered (a listed safety-critical one must have an L3 test).
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
`platform/`). For those, *the change most likely to break the invariant attributes no
requirement at all*, so requirement-scoped selection filters the check out of exactly that
change. Such a test **declares** its scope:
```yaml
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
```
and `--include-scope whole-repo` unions every declaring test into the selection. The CI
stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
annotation is the only step** — there is nothing to remember and nothing to hand-write.
Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
full rationale and the incidents behind it.
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
-56
View File
@@ -47,62 +47,6 @@ conf_dirs_for_requirements() {
done < <(conf_all_test_files) | sort -u
}
# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
# A test may also declare the SCOPE of the invariant it asserts:
#
# metadata:
# annotations:
# conformance.home.k3s/scope: "whole-repo"
#
# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
# ------------------------------------------------------------------------
# The merge gate runs the conformance subset tagged with the requirements a change
# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
# NOTHING. For those, the change most likely to break the invariant attributes no requirement
# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
# Three real incidents came out of that blind spot; each is recorded in the header of the
# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
# bulk-tier-distinct-from-replicated).
#
# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
# only when some unrelated requirement happens to be touched. The scope annotation replaces
# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
# every test carrying it into the selection, so adding the annotation is the only step.
#
# TWO things follow from the annotation, and both are deliberate:
# 1. the test is ALWAYS selected (union with the requirement-scoped subset), and
# 2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
# A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
# it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
# stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
# scope named in a test's header prose is documentation, not a declaration (the same
# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
conf_dirs_for_scope() {
local scope="$1" f
[ -n "$scope" ] || return 0
while IFS= read -r f; do
[ -n "$f" ] || continue
if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
dirname "$f"
fi
done < <(conf_all_test_files) | sort -u
}
# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
conf_dirs_for_scopes() {
local scopes="$1" s
for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
}
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
+2 -41
View File
@@ -467,14 +467,7 @@ def git_sha(repo: Path) -> str:
capture_output=True, text=True, check=True).stdout.strip()
except (subprocess.CalledProcessError, OSError):
return "unknown"
# EXCLUDE conformance/runs/. It is TRACKED, so `--record` writing ledger.jsonl dirtied the
# tree and every LATER run in the same session was stamped `-dirty` — recording one test
# poisoned the next, and check.sh refuses to honour a dirty proven row. That cost a second
# live fault-injection run on prod to work around. The docstring above already states the
# intent for the untracked runs/<id>/ half of this same artifact; the ledger is the tracked
# half. Mirrors status.py's IMPL_EXCLUDE_PREFIXES, which lists this path for the same reason.
dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--",
".", ":(exclude)conformance/runs/"],
dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
capture_output=True).returncode != 0
return f"{sha}-dirty" if dirty else sha
@@ -489,36 +482,6 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
return "unknown"
def context_env() -> str:
"""The env a run TARGETED, derived from the live kube context — never guessed.
WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
(`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
a proof that lies about where it was proven is worse than no proof, because the ledger's
whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
name: an unreadable context must not silently attribute a proof to an environment.
"""
try:
p = subprocess.run(["kubectl", "config", "current-context"],
capture_output=True, text=True, timeout=15)
except (OSError, subprocess.TimeoutExpired):
return "unknown"
if p.returncode != 0:
return "unknown"
ctx = (p.stdout or "").strip()
if not ctx:
return "unknown"
# contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
# unexpected context is visible in the ledger rather than being coerced into a known name.
return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
def server_kube_version() -> str:
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
for args in (["kubectl", "version", "-o", "json"],
@@ -913,9 +876,7 @@ def main() -> int:
help=f"append-only ledger (default {DEFAULT_LEDGER})")
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
help="env the run targeted; derived from the kube context when "
"unset (never defaulted to a cluster name)")
ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
ap.add_argument("--chainsaw-version", default=None,
help="override the versions.env pin (recorded verbatim)")
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
+8 -90
View File
@@ -19,30 +19,6 @@ Exit 0 iff every selected offline check passes.
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
carries; a dir with neither is reported SKIPPED (never a pass).
DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope)
----------------------------------------------------------------------
Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by
default, does not fail the run (a scaffold with no check script, an audit whose subject
subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the
checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages
mapped rc 3 to the GATE's rc 3 with the words "not a pass":
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
summary "- subst-safety: **SKIPPED** (nothing to audit)"
if [ "$rc" -eq 0 ]; then rc=3; fi
A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has
VANISHED — which is a louder problem than the check failing, not a quieter one. So:
--unwired-scope SCOPE a test DECLARING `<prefix>/scope: SCOPE` whose check SKIPS (rc 3,
or no check script at all) makes this runner exit 3 instead of 0.
Hard failures still win (exit 1); rc 3 never masks one.
Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a
scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
instead of once per hand-written stage.
"""
from __future__ import annotations
@@ -60,24 +36,6 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
def scope_of(test_yaml: Path, annotation: str) -> str:
"""The scope a test DECLARES, or "" when it declares none.
Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key,
so a scope named in the test's header prose is documentation, not a declaration — the
same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs.
Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree.
"""
if not test_yaml.exists():
return ""
rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$")
for line in test_yaml.read_text().splitlines():
m = rx.match(line)
if m:
return m.group(1)
return ""
def layer_of(d: Path) -> str:
"""Layer for the JUnit properties, from the subtree the test lives in."""
rel = str(d)
@@ -101,38 +59,25 @@ def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--junit", type=Path, required=True)
ap.add_argument("--label-prefix", default="conformance.home.k3s")
ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope",
help="annotation key a test uses to DECLARE its invariant's scope")
ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE",
help="a test declaring this scope that SKIPS is UNWIRED, not green "
"(makes this runner exit 3); repeatable")
ap.add_argument("--summary-out", type=Path, default=None,
help="write one Markdown line per case here, for the CI step summary")
ap.add_argument("dirs", nargs="+", type=Path)
args = ap.parse_args()
strict_scopes = set(args.unwired_scope)
cases = [] # (name, reqs, dir, rc, output, scope)
cases = [] # (name, reqs, dir, rc, output)
for d in args.dirs:
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
if check is None:
cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
scope))
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
continue
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True
)
out = (proc.stdout + proc.stderr).strip()
cases.append((name, reqs, d, proc.returncode, out, scope))
cases.append((name, reqs, d, proc.returncode, out))
failures = sum(1 for c in cases if c[3] not in (0, 3))
skipped = sum(1 for c in cases if c[3] == 3)
# A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring).
unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes]
suite = ET.Element(
"testsuite",
@@ -144,8 +89,7 @@ def main() -> int:
"skipped": str(skipped),
},
)
summary_lines = []
for name, reqs, d, rc, out, scope in cases:
for name, reqs, d, rc, out in cases:
tc = ET.SubElement(
suite,
"testcase",
@@ -156,33 +100,17 @@ def main() -> int:
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
ET.SubElement(props, "property", {"name": "scope", "value": scope})
if rc == 3:
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
elif rc != 0:
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
fail.text = out
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
mark = f" scope={scope}" if scope else ""
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
f"({','.join(reqs) or 'untagged'}){mark}",
f"({','.join(reqs) or 'untagged'})",
file=sys.stderr)
if rc not in (0, 3):
print(out, file=sys.stderr)
# One step-summary line per case, naming what ran — the generated replacement for the
# hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried.
tag = f" ({scope})" if scope else ""
if rc == 0:
verdict = "**passed**"
elif rc == 3:
verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes
else "**SKIPPED** (nothing to audit)")
else:
verdict = f"**FAILED** (exit {rc})"
summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}")
if rc == 3 and scope in strict_scopes:
print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — "
f"UNWIRED, NOT GREEN. {out}", file=sys.stderr)
root = ET.Element(
"testsuites",
@@ -198,19 +126,9 @@ def main() -> int:
args.junit.parent.mkdir(parents=True, exist_ok=True)
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
args.junit.write_text(xml)
if args.summary_out is not None:
args.summary_out.parent.mkdir(parents=True, exist_ok=True)
args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else ""))
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), "
f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr)
if failures:
return 1
if unwired:
print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): "
+ ", ".join(c[0] for c in unwired), file=sys.stderr)
return 3
return 0
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
f"{args.junit}", file=sys.stderr)
return 1 if failures else 0
if __name__ == "__main__":
+12 -84
View File
@@ -10,12 +10,6 @@
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
# ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
# # UNION: the tagged subset PLUS every test that
# # DECLARES `conformance.home.k3s/scope: whole-repo`.
# # A whole-repo invariant must run on EVERY gate
# # invocation — see lib/common.sh's scope block for
# # why, and why a SKIP of one is exit 3, not green.
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
# # requirement, is it implemented, is it actually
@@ -24,11 +18,6 @@
# # (runs/ledger.jsonl) actually proves it — see
# # lib/status.py. No ledger row => not-derivable.
# ./run.sh --lint # structural-validate test defs (no cluster)
# ./run.sh --print-selection # print the test dirs the CURRENT selection resolves
# # to, one per line, and exit. Runs no check. This is
# # how a test can PROVE the selection includes it
# # (tests/l0-static/whole-repo-invariants-wired) without
# # executing the suite and recursing into itself.
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
# ./run.sh --require-all # phase-exit: every requirement must be covered
@@ -90,7 +79,6 @@ ORIG_ARGS=("$@")
LAYER="all"
REQS=""
INCLUDE_SCOPES="" # --include-scope: scopes unioned into the selection (see lib/common.sh)
MODE="run" # run | coverage-only | status | lint
STRICT_COVERAGE=0
REQUIRE_ALL=0
@@ -134,11 +122,9 @@ while [ $# -gt 0 ]; do
case "$1" in
--layer) LAYER="$2"; shift 2 ;;
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
--include-scope) INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
--coverage-only) MODE="coverage-only"; shift ;;
--status) MODE="status"; shift ;;
--lint) MODE="lint"; shift ;;
--print-selection) MODE="print-selection"; shift ;;
--strict-coverage) STRICT_COVERAGE=1; shift ;;
--require-all) REQUIRE_ALL=1; shift ;;
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
@@ -201,7 +187,7 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
# to the real ledger; operators should never set it.
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
--run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
@@ -239,7 +225,7 @@ gen_status() {
--registry "${CONF_ROOT}/requirements/registry.yaml" \
--tests "${CONF_ROOT}/tests" \
--out "${STATUS_OUT}" \
${CONF_ENV:+--env "$CONF_ENV"} \
--env "${CONF_ENV:-dev}" \
"$@"
}
@@ -265,38 +251,11 @@ lint_tests() {
}
# --- Select which test directories Chainsaw should run --------------------------------
# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
#
# (--requirements ∩ --layer) ∪ (--include-scope ∩ --layer)
#
# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
# requirements a change touches, because the change most likely to break it touches none of
# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
# construction — it can only ADD tests to a run, never remove one.
select_dirs() {
{
requirement_scoped_dirs
# The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
# L2 test into an offline run. With no --layer (all), nothing is bounded away.
if [ -n "${INCLUDE_SCOPES}" ]; then
if [ "${LAYER}" != all ]; then
comm -12 \
<(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
<(conf_dirs_for_layer "${LAYER}")
else
conf_dirs_for_scopes "${INCLUDE_SCOPES}"
fi
fi
} | sort -u
}
# The pre-existing requirement/layer selection, unchanged:
# --requirements AND --layer together => intersection (tests carrying the tag
# AND living in that layer subtree). Either alone keeps its prior meaning, so
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
requirement_scoped_dirs() {
# --requirements AND --layer together => intersection (tests carrying the tag
# AND living in that layer subtree). Either alone keeps its prior meaning, so
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
comm -12 \
<(conf_dirs_for_requirements "${REQS}") \
@@ -437,12 +396,6 @@ layers_are_offline_only() {
return 0
}
# Markdown one-liner per executed offline check, for the CI step summary. Written next to
# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
run_offline_checks() {
local -a dirs
mapfile -t dirs < <(offline_selected_dirs)
@@ -450,30 +403,14 @@ run_offline_checks() {
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
return 0
fi
# Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
# report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
local -a scope_args=() s
for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
[ -n "${INCLUDE_SCOPES}" ] && log " unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
--scope-annotation "${CONF_SCOPE_ANNOTATION}" \
--summary-out "${OFFLINE_SUMMARY_OUT}" \
${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
}
###############################################################################
# Main
###############################################################################
# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
# writing nothing and running no check. A test that asserts "the gate's selection cannot
# filter me out" needs to ask that question without executing the suite it is part of.
if [ "${MODE}" = "print-selection" ]; then
select_dirs
exit 0
fi
log "Coverage report first (always)"
gen_coverage
@@ -503,26 +440,17 @@ esac
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
# exactly as the deleted per-check stages did.
OFFLINE_RC=0
run_offline_checks || OFFLINE_RC=$?
if layers_are_offline_only; then
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
case "${OFFLINE_RC}" in
0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
*) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
esac
if [ "${OFFLINE_RC}" -eq 0 ]; then
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
else
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
fi
exit "${OFFLINE_RC}"
fi
if [ "${OFFLINE_RC}" -eq 3 ]; then
log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
exit 3
fi
if [ "${OFFLINE_RC}" -ne 0 ]; then
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
exit "${OFFLINE_RC}"
@@ -1,12 +0,0 @@
<?xml version="1.0" ?>
<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
<testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
<testcase classname="L1:R-NET-05,R-NET-06,R-NET-07" name="egress-class-render">
<properties>
<property name="requirements" value="R-NET-05,R-NET-06,R-NET-07"/>
<property name="layer" value="L1"/>
<property name="path" value="/work/.claude/conf-clone/conformance/tests/l1-rendering/egress-class-render"/>
</properties>
</testcase>
</testsuite>
</testsuites>
@@ -1,7 +0,0 @@
[run.sh] Coverage report first (always)
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
[offline] L1 PASS egress-class-render (R-NET-05,R-NET-06,R-NET-07)
[offline] wrote JUnit (1 case(s), 0 failure(s)) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml
[run.sh] Done (offline l1). JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
@@ -1,4 +0,0 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="vpn-killswitch" classname="" time="174.632779" />
</testsuite>
</testsuites>
@@ -1,237 +0,0 @@
[run.sh] Coverage report first (always)
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
Version: 0.2.12
Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-light'
- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 2
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/vpn-killswitch
=== PAUSE chainsaw/vpn-killswitch
=== CONT chainsaw/vpn-killswitch
l.go:52: | 01:44:38 | vpn-killswitch | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-credible-bird
l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | BEGIN |
l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
kubectl label namespace "$NAMESPACE" \
pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/warn=privileged \
pod-security.kubernetes.io/audit=privileged \
--overwrite
echo "NS_PRIVILEGED_OK"
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
namespace/chainsaw-credible-bird labeled
NS_PRIVILEGED_OK
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
get_ip() {
kubectl -n "$ns" exec "$1" -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
}
wan="" vpn=""
for i in $(seq 1 12); do
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
sleep 5
done
echo "wan=$wan vpn=$vpn"
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
echo "TUNNEL_HEALTHY_OK"
l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
wan=136.51.115.252 vpn=158.173.241.11
TUNNEL_HEALTHY_OK
l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | END |
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | BEGIN |
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | RUN | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CREATE | OK | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | DONE | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | RUN | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | DONE | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -u
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
for i in $(seq 1 8); do
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$out" ]; then
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "attempt $i: vpn-class blackholed (no response), good"
sleep 3
done
echo "NO_LEAK_OK"
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
attempt 1: vpn-class blackholed (no response), good
attempt 2: vpn-class blackholed (no response), good
attempt 3: vpn-class blackholed (no response), good
attempt 4: vpn-class blackholed (no response), good
attempt 5: vpn-class blackholed (no response), good
attempt 6: vpn-class blackholed (no response), good
attempt 7: vpn-class blackholed (no response), good
attempt 8: vpn-class blackholed (no response), good
NO_LEAK_OK
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
echo "internet_ip=$out"
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
echo "INTERNET_UNAFFECTED_OK"
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
internet_ip=136.51.115.252
INTERNET_UNAFFECTED_OK
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | END |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | BEGIN |
l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
# Distinct Job name (tunnel-restart) so it never collides with the
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
# valid YAML at column 0 at runtime.
cat <<'JOB' | kubectl -n "$ns" apply -f -
apiVersion: batch/v1
kind: Job
metadata:
name: tunnel-restart
labels: { app: vpn-killswitch-fault }
spec:
backoffLimit: 3
activeDeadlineSeconds: 90
ttlSecondsAfterFinished: 600
template:
metadata:
labels: { app: vpn-killswitch-fault }
spec:
restartPolicy: Never
hostPID: true
nodeSelector:
homelab.keithjackson.dev/egress-gateway: "true"
tolerations:
- operator: Exists
containers:
- name: nsenter
image: busybox:1.37.0
securityContext:
privileged: true
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
JOB
# Block teardown until the restart actually completed on the host — a
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
# or timeout, which (set -e) fails the step and surfaces the problem.
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
echo "TUNNEL_SELF_HEAL_OK"
l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
job.batch/tunnel-restart created
job.batch/tunnel-restart condition met
TUNNEL_SELF_HEAL_OK
l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | END |
l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | BEGIN |
l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
vpn=""
for i in $(seq 1 24); do
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
vpn=""
sleep 5
done
echo "recovered_vpn=$vpn wan=$wan"
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
echo "TUNNEL_RECOVERED_OK"
l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | LOG |
=== STDOUT
recovered_vpn=158.173.241.11 wan=136.51.115.252
TUNNEL_RECOVERED_OK
l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | DONE |
l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | END |
l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | BEGIN |
l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | DELETE | OK | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | END |
l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | BEGIN |
l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-credible-bird/egress-vpn
l.go:52: | 01:46:56 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-credible-bird/egress-internet
l.go:52: | 01:47:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | END |
l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-credible-bird
l.go:52: | 01:47:33 | vpn-killswitch | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/vpn-killswitch (174.63s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
[run.sh] HEAVY pass: no heavy tests selected.
merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
@@ -1,4 +0,0 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="vpn-killswitch" classname="" time="173.473422" />
</testsuite>
</testsuites>
@@ -1,237 +0,0 @@
[run.sh] Coverage report first (always)
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
Version: 0.2.12
Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-light'
- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 2
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/vpn-killswitch
=== PAUSE chainsaw/vpn-killswitch
=== CONT chainsaw/vpn-killswitch
l.go:52: | 01:49:24 | vpn-killswitch | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-beloved-toad
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | BEGIN |
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
kubectl label namespace "$NAMESPACE" \
pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/warn=privileged \
pod-security.kubernetes.io/audit=privileged \
--overwrite
echo "NS_PRIVILEGED_OK"
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
namespace/chainsaw-beloved-toad labeled
NS_PRIVILEGED_OK
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | RUN | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CREATE | OK | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | APPLY | DONE | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | RUN | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | ASSERT | DONE | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
get_ip() {
kubectl -n "$ns" exec "$1" -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
}
wan="" vpn=""
for i in $(seq 1 12); do
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
sleep 5
done
echo "wan=$wan vpn=$vpn"
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
echo "TUNNEL_HEALTHY_OK"
l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | LOG |
=== STDOUT
wan=136.51.115.252 vpn=158.173.241.11
TUNNEL_HEALTHY_OK
l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | SCRIPT | DONE |
l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | TRY | END |
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | BEGIN |
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | RUN | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CREATE | OK | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | APPLY | DONE | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | RUN | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | ASSERT | DONE | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -u
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
for i in $(seq 1 8); do
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$out" ]; then
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
fi
echo "attempt $i: vpn-class blackholed (no response), good"
sleep 3
done
echo "NO_LEAK_OK"
l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
attempt 1: vpn-class blackholed (no response), good
attempt 2: vpn-class blackholed (no response), good
attempt 3: vpn-class blackholed (no response), good
attempt 4: vpn-class blackholed (no response), good
attempt 5: vpn-class blackholed (no response), good
attempt 6: vpn-class blackholed (no response), good
attempt 7: vpn-class blackholed (no response), good
attempt 8: vpn-class blackholed (no response), good
NO_LEAK_OK
l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
echo "internet_ip=$out"
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
echo "INTERNET_UNAFFECTED_OK"
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
internet_ip=136.51.115.252
INTERNET_UNAFFECTED_OK
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | TRY | END |
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | BEGIN |
l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
# Distinct Job name (tunnel-restart) so it never collides with the
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
# valid YAML at column 0 at runtime.
cat <<'JOB' | kubectl -n "$ns" apply -f -
apiVersion: batch/v1
kind: Job
metadata:
name: tunnel-restart
labels: { app: vpn-killswitch-fault }
spec:
backoffLimit: 3
activeDeadlineSeconds: 90
ttlSecondsAfterFinished: 600
template:
metadata:
labels: { app: vpn-killswitch-fault }
spec:
restartPolicy: Never
hostPID: true
nodeSelector:
homelab.keithjackson.dev/egress-gateway: "true"
tolerations:
- operator: Exists
containers:
- name: nsenter
image: busybox:1.37.0
securityContext:
privileged: true
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
JOB
# Block teardown until the restart actually completed on the host — a
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
# or timeout, which (set -e) fails the step and surfaces the problem.
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
echo "TUNNEL_SELF_HEAL_OK"
l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | LOG |
=== STDOUT
job.batch/tunnel-restart created
job.batch/tunnel-restart condition met
TUNNEL_SELF_HEAL_OK
l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | SCRIPT | DONE |
l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | FINALLY | END |
l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | BEGIN |
l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
ns="$NAMESPACE"
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
vpn=""
for i in $(seq 1 24); do
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
vpn=""
sleep 5
done
echo "recovered_vpn=$vpn wan=$wan"
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
echo "TUNNEL_RECOVERED_OK"
l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | LOG |
=== STDOUT
recovered_vpn=158.173.241.11 wan=136.51.115.252
TUNNEL_RECOVERED_OK
l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT | DONE |
l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY | END |
l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | BEGIN |
l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | DELETE | OK | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak  | CLEANUP | END |
l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | BEGIN |
l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-beloved-toad/egress-vpn
l.go:52: | 01:51:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | DELETE | OK | v1/Pod @ chainsaw-beloved-toad/egress-internet
l.go:52: | 01:52:10 | vpn-killswitch | given-healthy-tunnel-and-two-workloads  | CLEANUP | END |
l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-beloved-toad
l.go:52: | 01:52:18 | vpn-killswitch | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/vpn-killswitch (173.47s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
[run.sh] HEAVY pass: no heavy tests selected.
merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
@@ -1,4 +0,0 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="dr-restore" classname="" time="150.698290" />
</testsuite>
</testsuites>
@@ -1,672 +0,0 @@
[run.sh] Coverage report first (always)
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
[run.sh] LIGHT pass: no tests selected.
[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
Version: 0.2.12
Loading config (/work/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-heavy'
- ReportPath '/work/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 1
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/dr-restore
=== PAUSE chainsaw/dr-restore
=== CONT chainsaw/dr-restore
l.go:52: | 02:15:05 | dr-restore | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-glorious-sheep
l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster  | TRY | BEGIN |
l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
# ---- helpers -------------------------------------------------------
# Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
# fail loudly; an absent API is a genuine precondition gap and may skip.
die_or_skip() {
case "$2" in
*orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
echo "$2" >&2
exit 1 ;;
*)
echo "SKIP: $1 not available on this cluster -- $2"
echo DR_RESTORE_SKIPPED_OK
exit 0 ;;
esac
}
ip2int() {
_q="$(echo "$1" | tr '.' ' ')"
set -- $_q
[ $# -eq 4 ] || { echo -1; return 0; }
echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
}
# in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
in_cidr() {
case "$2" in *"/"*) ;; *) return 1 ;; esac
_net="${2%/*}"; _len="${2#*/}"
case "$_len" in ''|*[!0-9]*) return 1 ;; esac
[ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
_i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
[ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
[ "$_len" -eq 0 ] && return 0
_d=1; _s=0
while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
[ $(( _i / _d )) -eq $(( _n / _d )) ]
}
# ---- Gate 1: the replicated storage tier the DR-critical data lives on --
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
echo DR_RESTORE_SKIPPED_OK
exit 0
fi
# ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
echo DR_RESTORE_SKIPPED_OK
exit 0
fi
echo "backup storage locations:"
echo "$BSL_LINES"
VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
# ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
# File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
# deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
# ready node-agent no volume data can be captured, so the drill is not runnable.
VREADY=0; NAREADY=0
for i in $(seq 1 12); do
VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
sleep 10
done
if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
echo DR_RESTORE_SKIPPED_OK
exit 0
fi
# ---- Pick the BSL this drill will use: prefer the default, and it MUST be
# Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
# gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
# wait for first-validation on a cold start), it does not skip. --------
BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
for i in $(seq 1 12); do
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
if [ -n "$CAND" ]; then
BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
break
fi
echo "poll $i: no Available BackupStorageLocation yet"
sleep 10
done
if [ -z "$BSL_NAME" ]; then
echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
echo " The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
echo "$BSL_LINES" >&2
exit 1
fi
echo "using BSL ${BSL_NS}/${BSL_NAME} bucket=${BSL_BUCKET} endpoint=${BSL_URL}"
# ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
if [ -z "$BSL_URL" ]; then
echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
echo " the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
echo " test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
exit 1
fi
HOST="${BSL_URL#*://}" # strip scheme
HOST="${HOST%%/*}" # strip path
HOST="${HOST%%:*}" # strip port
echo "backup endpoint host: $HOST"
case "$HOST" in
localhost|*.svc|*.svc.cluster.local|*.cluster.local)
echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
echo " An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
exit 1 ;;
esac
# Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
# off-cluster property -- that is a failure, never a silent pass.
case "$HOST" in
*[!0-9.]*)
# Endpoint given as a name. Try the resolvers that may exist in the
# runner image, in order; if none is available or none resolves, the
# test FAILS below rather than guessing.
EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
[ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
[ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
*) EPIP="$HOST" ;;
esac
if [ -z "$EPIP" ]; then
echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
echo " the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
exit 1
fi
echo "backup endpoint resolves to: $EPIP"
# ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
for a in $NODE_IPS; do
if [ "$a" = "$EPIP" ]; then
echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
exit 1
fi
done
echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
for a in $SVC_IPS; do
if [ "$a" = "$EPIP" ]; then
echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
exit 1
fi
done
echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
for a in $POD_IPS; do
if [ "$a" = "$EPIP" ]; then
echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
exit 1
fi
done
echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
for c in $POD_CIDRS; do
case "$c" in *:*) continue ;; esac # IPv6 prefixes are out of scope here
if in_cidr "$EPIP" "$c"; then
echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
exit 1
fi
done
echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
# Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
# runner may not read): the checks above are the authoritative ones.
SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
if [ -n "$SVC_CIDRS" ]; then
for c in $SVC_CIDRS; do
case "$c" in *:*) continue ;; esac
if in_cidr "$EPIP" "$c"; then
echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
exit 1
fi
done
echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
else
echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
fi
echo "OFF_CLUSTER_TARGET_OK bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster  | SCRIPT | LOG |
=== STDOUT
backup storage locations:
velero default Available aws true https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com k3s-prod-backup
poll 1: velero readyReplicas=1 node-agent numberReady=3
using BSL velero/default bucket=k3s-prod-backup endpoint=https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
backup endpoint host: 7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
backup endpoint resolves to: 172.64.66.1
endpoint is not any node address (checked 6 node addresses)
endpoint is not any Service ClusterIP (checked 82)
endpoint is not any Pod IP (checked 197)
endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
endpoint is outside every ServiceCIDR (10.43.0.0/16 )
OFF_CLUSTER_TARGET_OK bsl=velero/default endpoint=172.64.66.1 bucket=k3s-prod-backup
l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster  | SCRIPT | DONE |
l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster  | TRY | END |
l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled  | TRY | BEGIN |
l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled  | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
SCHED="$(kubectl get schedules.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
# Distinguish "Velero absent" (skip, in lock-step with step 1) from
# "Velero present but nothing scheduled" (a real R-STATE-03 defect).
if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
echo DR_SCHEDULE_SKIPPED_OK
exit 0
fi
echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
echo " R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
exit 1
fi
echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
echo DR_SCHEDULE_SKIPPED_OK
exit 0
fi
echo "velero schedules:"
echo "$SCHED"
# A Schedule that exists but is paused/disabled backs nothing up.
if echo "$SCHED" | grep -q 'phase=Enabled'; then
echo BACKUPS_ARE_SCHEDULED_OK
else
echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
exit 1
fi
l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled  | SCRIPT | LOG |
=== STDOUT
velero schedules:
velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T04:01:06Z
velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T03:00:03Z
BACKUPS_ARE_SCHEDULED_OK
l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled  | SCRIPT | DONE |
l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled  | TRY | END |
l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | TRY | BEGIN |
l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -eu
NS="$NAMESPACE"
SRC="${NS}-dr-src" # scratch source namespace (destroyed by design)
DST="${NS}-dr-dst" # clean restore target (must not pre-exist)
BK="${NS}-dr-backup"
RS="${NS}-dr-restore"
OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
# ---- Re-derive the preconditions (steps do not share shell state). Every
# skip below mirrors step 1 exactly, so the whole test skips or runs
# as a unit -- it can never half-run. -----------------------------
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
fi
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
if [ -z "$CAND" ]; then
echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
echo DR_RESTORE_SKIPPED_OK; exit 0
fi
VNS="$(echo "$CAND" | cut -d' ' -f1)"
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
echo DR_RESTORE_SKIPPED_OK; exit 0
fi
echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
# A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
# be free. If either is lying around from an aborted run, remove it before we
# claim "clean" -- and fail loudly if it will not go, because a half-deleted
# namespace would silently invalidate the destroy proof below.
kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
# ============================ GIVEN ==============================
# A DR-critical stateful workload with known data on the replicated tier.
CANARY="dr-canary-$(date +%s%N)"
kubectl create namespace "$SRC"
kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
kubectl -n "$SRC" apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dr-data
labels:
conformance.home.k3s/created-by: ${NS}
spec:
accessModes: [ReadWriteOnce]
storageClassName: longhorn
resources:
requests:
storage: 1Gi
EOF
kubectl -n "$SRC" apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: dr-workload
labels:
app: dr-workload
conformance.home.k3s/created-by: ${NS}
spec:
restartPolicy: Never
containers:
- name: app
image: busybox:1.36.1
command: ["sh", "-c", "sleep 7200"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: dr-data
EOF
kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
[ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
echo "original in-cluster PersistentVolume: $ORIG_PV"
# ======================= WHEN (part 1): back up off-cluster =======
# Created as a Velero Backup CR -- exactly what `velero backup create` does.
# defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
# the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
# ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
# own GC still removes this drill's objects from the bucket.
kubectl apply -f - <<EOF
apiVersion: velero.io/v1
kind: Backup
metadata:
name: ${BK}
namespace: ${VNS}
labels:
conformance.home.k3s/created-by: ${NS}
spec:
includedNamespaces: [${SRC}]
storageLocation: ${BSL_NAME}
snapshotVolumes: false
defaultVolumesToFsBackup: true
ttl: 2h0m0s
EOF
BPHASE=""
for i in $(seq 1 90); do
BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
echo "poll $i: backup phase=${BPHASE:-<pending>}"
case "${BPHASE:-}" in
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
esac
sleep 10
done
if [ "${BPHASE:-}" != "Completed" ]; then
echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
exit 1
fi
echo "backup $BK Completed to ${BSL_NAME}"
# ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
# kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
-l velero.io/backup-name="$BK" \
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
echo "pod volume backups:"
echo "$PVB"
if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
echo " Resource-only backups cannot restore DR-critical DATA." >&2
exit 1
fi
# A line is "<name> <phase> <snapshotID>"; require Completed AND a
# non-empty snapshot ID (an empty ID means nothing reached the repository).
GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
if [ "${GOOD_PVB:-0}" -lt 1 ]; then
echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
echo " the volume data did not reach the off-cluster repository." >&2
exit 1
fi
echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
# ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
# Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
# underlying PersistentVolume (and with it the Longhorn replicas). After this
# there is no in-cluster source the restore could quietly read from.
kubectl delete namespace "$SRC" --wait=true --timeout=420s
for i in $(seq 1 30); do
if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
echo "poll $i: waiting for namespace $SRC to disappear"
sleep 10
done
if kubectl get namespace "$SRC" >/dev/null 2>&1; then
echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
exit 1
fi
echo "source namespace destroyed"
# The longhorn StorageClass reclaims Delete, so the PV should already be gone;
# force it if a Retain policy (or a stuck finalizer) left it behind, because a
# surviving PV would make the restore prove nothing about off-cluster recovery.
kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
for i in $(seq 1 30); do
if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
sleep 10
done
if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
echo " An in-cluster copy of the data still exists, so a successful restore" >&2
echo " would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
exit 1
fi
echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
# Best-effort corroboration (the least-privilege runner may not read longhorn.io):
# the Longhorn Volume CR and its replicas should be gone too.
if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
exit 1
fi
echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
# ============================ RESTORE ============================
# Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
# the catalog's "restored into a clean state".
kubectl apply -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
name: ${RS}
namespace: ${VNS}
labels:
conformance.home.k3s/created-by: ${NS}
spec:
backupName: ${BK}
includedNamespaces: [${SRC}]
namespaceMapping:
${SRC}: ${DST}
existingResourcePolicy: none
EOF
RPHASE=""
for i in $(seq 1 90); do
RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
echo "poll $i: restore phase=${RPHASE:-<pending>}"
case "${RPHASE:-}" in
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
esac
sleep 10
done
if [ "${RPHASE:-}" != "Completed" ]; then
echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
exit 1
fi
echo "restore $RS Completed into $DST"
# ============================= THEN ==============================
# The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
# `restore-wait` init container that blocks until the node-agent has written the
# volume contents back, so Ready here already implies the data landed.
if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
kubectl -n "$DST" describe pod dr-workload >&2 || true
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
exit 1
fi
NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
[ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
if [ "$NEW_PV" = "$ORIG_PV" ]; then
echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
echo " That is an in-cluster survival, not an off-cluster restore." >&2
exit 1
fi
echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
echo "recovered='$GOT' expected='$CANARY'"
if [ "$GOT" != "$CANARY" ]; then
echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
exit 1
fi
echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | LOG |
=== STDOUT
velero ns=velero bsl=default src=chainsaw-glorious-sheep-dr-src dst=chainsaw-glorious-sheep-dr-dst
namespace/chainsaw-glorious-sheep-dr-src created
namespace/chainsaw-glorious-sheep-dr-src labeled
persistentvolumeclaim/dr-data created
pod/dr-workload created
pod/dr-workload condition met
seeded canary: dr-canary-1789352108935754215
original in-cluster PersistentVolume: pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9
backup.velero.io/chainsaw-glorious-sheep-dr-backup created
poll 1: backup phase=InProgress
poll 2: backup phase=InProgress
poll 3: backup phase=InProgress
poll 4: backup phase=Completed
backup chainsaw-glorious-sheep-dr-backup Completed to default
pod volume backups:
chainsaw-glorious-sheep-dr-backup-2gjrq Completed 979eed437233b45c7f85ba80074ab56b
volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
namespace "chainsaw-glorious-sheep-dr-src" deleted
source namespace destroyed
original PersistentVolume pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 is GONE -- no in-cluster copy of the data remains
(longhorn Volume CR pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 not present -- either deleted or not readable by this identity)
restore.velero.io/chainsaw-glorious-sheep-dr-restore created
poll 1: restore phase=InProgress
poll 2: restore phase=InProgress
poll 3: restore phase=InProgress
poll 4: restore phase=Completed
restore chainsaw-glorious-sheep-dr-restore Completed into chainsaw-glorious-sheep-dr-dst
pod/dr-workload condition met
restored PVC bound to PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
restored onto a NEW volume (pvc-f0c061e1-c080-49f2-a381-23da5c886df2 != pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9) -- the data came from outside the cluster
recovered='dr-canary-1789352108935754215' expected='dr-canary-1789352108935754215'
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
=== STDERR
Defaulted container "app" out of: app, restore-wait (init)
l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | DONE |
l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | TRY | END |
l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY | BEGIN |
l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -u
NS="$NAMESPACE"
SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
# Velero objects first: a DeleteBackupRequest makes Velero remove the backup
# DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
if [ -n "$VNS" ]; then
kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
kubectl apply -f - <<EOF || true
apiVersion: velero.io/v1
kind: DeleteBackupRequest
metadata:
name: ${BK}-delete
namespace: ${VNS}
labels:
conformance.home.k3s/created-by: ${NS}
spec:
backupName: ${BK}
EOF
for i in $(seq 1 18); do
kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
sleep 10
done
# Backstop if Velero could not process the request (the 2h ttl also GCs it).
kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
fi
echo "velero objects for $NS cleaned up (ns $VNS)"
fi
# Scratch namespaces (both, in case the run aborted at any point).
for n in "$DST" "$SRC"; do
kubectl delete namespace "$n" --ignore-not-found --wait=false || true
done
# Anything left labelled as ours (defensive; the two names above are the only
# namespaces this test ever creates).
kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
# Released PVs from either namespace's PVCs would otherwise leak storage.
for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
kubectl delete pv "$pv" --ignore-not-found --wait=false || true
echo "released leftover PersistentVolume: $pv"
done
echo "dr-restore teardown complete (best-effort; every object was created by this test)"
l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | LOG |
=== STDOUT
restore.velero.io "chainsaw-glorious-sheep-dr-restore" deleted from velero namespace
deletebackuprequest.velero.io/chainsaw-glorious-sheep-dr-backup-delete created
velero objects for chainsaw-glorious-sheep cleaned up (ns velero)
namespace "chainsaw-glorious-sheep-dr-dst" deleted
namespace "chainsaw-glorious-sheep-dr-dst" deleted
persistentvolume "pvc-f0c061e1-c080-49f2-a381-23da5c886df2" deleted
released leftover PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
dr-restore teardown complete (best-effort; every object was created by this test)
l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT | DONE |
l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY | END |
l.go:52: | 02:17:30 | dr-restore | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 02:17:30 | dr-restore | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-glorious-sheep
l.go:52: | 02:17:36 | dr-restore | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/dr-restore (150.70s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
@@ -1,13 +0,0 @@
<?xml version="1.0" ?>
<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
<testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
<testcase classname="L0:R-ENV-02,R-STATE-03" name="placeholder-guard-coupling">
<properties>
<property name="requirements" value="R-ENV-02,R-STATE-03"/>
<property name="layer" value="L0"/>
<property name="path" value="/work/conformance/tests/l0-static/placeholder-guard-coupling"/>
<property name="scope" value=""/>
</properties>
</testcase>
</testsuite>
</testsuites>
@@ -1,7 +0,0 @@
[run.sh] Coverage report first (always)
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
[offline] L0 PASS placeholder-guard-coupling (R-ENV-02,R-STATE-03)
[offline] wrote JUnit (1 case(s), 0 failure(s), 0 skip(s), 0 unwired) -> /work/conformance/_artifacts/report.junit.xml
[run.sh] Done (offline l0). JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
@@ -1,6 +0,0 @@
[run.sh] Coverage report first (always)
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
Install the pinned binary or run via the in-cluster Job (incluster/).
@@ -1,4 +0,0 @@
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
<testcase name="node-loss" classname="" time="115.855688" />
</testsuite>
</testsuites>
@@ -1,232 +0,0 @@
[run.sh] Coverage report first (always)
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
[run.sh] LIGHT pass: no tests selected.
[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
Version: 0.2.12
Loading config (/work/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-heavy'
- ReportPath '/work/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 1
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
Loading values...
Running tests...
=== RUN chainsaw
=== PAUSE chainsaw
=== CONT chainsaw
=== RUN chainsaw/node-loss
=== PAUSE chainsaw/node-loss
=== CONT chainsaw/node-loss
l.go:52: | 05:22:26 | node-loss | @chainsaw  | CREATE | OK | v1/Namespace @ chainsaw-advanced-boxer
l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | TRY | BEGIN |
l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -euo pipefail
NS="$NAMESPACE"
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
echo NODE_LOSS_SKIPPED_OK
exit 0
fi
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
READY="$(kubectl get nodes \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
| grep -c '^True$' || true)"
echo "ready nodes: ${READY}"
if [ "${READY:-0}" -lt 3 ]; then
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
echo NODE_LOSS_SKIPPED_OK
exit 0
fi
CANARY="node-loss-canary-$(date +%s%N)"
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
# writer pod that writes the canary and flushes it to the replicated volume. --
kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: survivor
spec:
accessModes: [ReadWriteOnce]
storageClassName: longhorn
resources:
requests:
storage: 1Gi
EOF
apply_pod() {
kubectl -n "$NS" apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: $1
spec:
restartPolicy: Never
containers:
- name: app
image: busybox:1.36.1
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: survivor
EOF
}
apply_pod writer
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
# The victim node is the one the attached volume currently lives on.
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
echo "victim node (currently hosts the attached volume): $VICTIM"
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
# runner is least-privilege and may not have longhorn.io read rights) — the
# data-availability assertion below is the authoritative proof, not this.
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
if [ -n "$PVNAME" ]; then
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
fi
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
# Label it with THIS test's unique namespace so `finally` can always find and
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
# pods on the node are never touched.)
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
kubectl cordon "$VICTIM"
echo "cordoned + labelled victim node: $VICTIM"
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
echo "evicted writer off the victim node; volume detached from $VICTIM"
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
# the volume there from its replicas on the other nodes. --
apply_pod reader
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
if [ "$READER_NODE" = "$VICTIM" ]; then
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
exit 1
fi
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
echo "recovered='$GOT' expected='$CANARY'"
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
echo "volume available + data intact from a surviving node"
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
# (bounded) for robustness to settle back to available (healthy|degraded — both
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
# Non-fatal for the same least-privilege reason as the pre-fault log.
if [ -n "$PVNAME" ]; then
for i in $(seq 1 30); do
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
echo "post-fault robustness[$i]=$ROB"
case "$ROB" in healthy|degraded) break ;; esac
sleep 10
done
fi
echo NODE_LOSS_VOLUME_AVAILABLE_OK
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT | LOG |
=== STDOUT
ready nodes: 3
persistentvolumeclaim/survivor created
pod/writer created
pod/writer condition met
wrote canary: node-loss-canary-1789795346371286043
victim node (currently hosts the attached volume): k3s-prod-2
pre-fault robustness=healthy replicas=3
node/k3s-prod-2 labeled
node/k3s-prod-2 cordoned
cordoned + labelled victim node: k3s-prod-2
pod "writer" deleted from chainsaw-advanced-boxer namespace
evicted writer off the victim node; volume detached from k3s-prod-2
pod/reader created
pod/reader condition met
reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
volume available + data intact from a surviving node
post-fault robustness[1]=healthy
NODE_LOSS_VOLUME_AVAILABLE_OK
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT | DONE |
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | TRY | END |
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | FINALLY | BEGIN |
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD | RUN |
=== COMMAND
/usr/bin/sh -c set -uo pipefail
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
kubectl uncordon "${n#node/}" || true
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
echo "restored node to service: ${n#node/}"
done
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT | LOG |
=== STDOUT
node/k3s-prod-2 uncordoned
node/k3s-prod-2 unlabeled
restored node to service: k3s-prod-2
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT | DONE |
l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD | RUN |
=== COMMAND
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT | LOG |
=== STDOUT
pod "reader" deleted from chainsaw-advanced-boxer namespace
persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
cleanup best-effort; chainsaw namespace teardown finalizes
l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT | DONE |
l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | FINALLY | END |
l.go:52: | 05:23:38 | node-loss | @chainsaw  | CLEANUP | BEGIN |
l.go:52: | 05:23:38 | node-loss | @chainsaw  | DELETE | OK | v1/Namespace @ chainsaw-advanced-boxer
l.go:52: | 05:24:21 | node-loss | @chainsaw  | CLEANUP | END |
--- PASS: chainsaw (0.00s)
--- PASS: chainsaw/node-loss (115.86s)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
-6
View File
@@ -1,8 +1,2 @@
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}
@@ -1,128 +0,0 @@
# =============================================================================
# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
# survives every dashboard (nobody looks for a panel that was never built).
#
# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
# count(longhorn_volume_robustness) => (empty) 4 rules
# count({__name__=~"certmanager.*"}) => (empty) 3 rules
# count(hubble_drop_total) => (empty) 1 rule
# count by (job) (up) => 20 jobs, none Longhorn/cert-manager/Hubble
# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
# was the same defect and was fixed separately in a79693d.
#
# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
# WHEN each rule's PromQL is parsed for the metric names it selects,
# THEN (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
# claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
# the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
# platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
# storage alerts;
# (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
# being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
# both set, Longhorn's own NetworkPolicies drop the scrape unless
# networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
# kube-state-metrics' metricLabelsAllowlist names the label;
# (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
# shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
# check above can never see this failure: where the repo authors a Service and a
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
# actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
# named port `metrics`; no app Service defined one) and were registered in
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
# THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
# kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
# ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
# moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
# alert rule to reach a monitor, resolves by selector across the whole tree instead of by
# same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
# lack of subjects;
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
#
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
# signal that already existed. The register is empty now, and that is the outcome it was for.)
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
# into fiction.
#
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
# run it directly, and this Chainsaw Test wraps the SAME script.
#
# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
# TARGET exists, never that the exporter emits that exact SERIES NAME.
# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
# and did not exist at all on the deployed KubeVirt (corrected here to
# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
# that diffs alert metrics against /api/v1/label/__name__/values.
#
# Also not proven: that a target is UP. This is static — it asserts the
# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
# (R-OBS-02 / the L3 alerting test).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-metrics-have-scrape-targets
labels:
conformance.home.k3s/R-OBS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
# every scrape-enabling declaration that feeds them, which live in eight different components
# (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
# platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
# path-maps to that set.
#
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
# platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
# apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
# false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
# CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
# ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
# UNWIRED, never green) rather than silently green.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every metric name queried by every alert rule is claimed by a scrape target whose enabling
declaration is still present in this repo, every `up{job=...}` selector names a job a repo
monitor can actually produce, and the two gaps that remain are enumerated in registers that
can only shrink — so an alert can never again be authored, reviewed and merged against a
metric nothing scrapes.
spec:
steps:
- name: audit-every-alert-metric-against-the-configured-scrape-targets
description: >-
Parse every metric name out of every alerting rule, resolve each against the declared
scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
agreement, and ratchet both registers of declared gaps.
try:
- script:
content: bash ./check.sh
@@ -1,739 +0,0 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
# VPN)." — spec docs/10-observability-and-alerting.md)
#
# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE
# -----------------------------------------------------------------------------------------------
# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series
# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and
# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is
# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit
# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because
# nobody looks for a panel that was never built.
#
# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert:
# count(longhorn_volume_robustness) => (empty) LonghornVolumeDegraded/Faulted/
# CapacityHigh, LonghornNodeStorageHigh
# count({__name__=~"certmanager.*"}) => (empty) CertificateExpiringSoon/
# ExpiryImminent/NotReady
# count(hubble_drop_total) => (empty) HubbleDeniedFlowSpike
# count by (job) (up) => 20 jobs, none Longhorn, cert-manager or Hubble
# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS
# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no
# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed
# separately in a79693d.
#
# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a
# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is
# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule
# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different
# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is
# what the owner table below does.
#
# GIVEN every alerting rule in every PrometheusRule in the repo,
# WHEN each rule's PromQL is parsed for the metric names it selects,
# THEN (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the
# owner's `require` regexes still match its file — so deleting the enablement from
# platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four
# storage alerts;
# (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be
# defeated without being deleted (cert-manager's chart `fail`s if podmonitor and
# servicemonitor are both enabled; an `enabled: false` inside the exporter's own block);
# (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only);
# (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to
# this — by requiring that where this repo authors a Service and a ServiceMonitor in the
# SAME file, the monitor's endpoint port is a port that Service actually defines;
# violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only);
# AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo`
# so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure.
#
# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as
# alert-rules-unit-tested faced:
# * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need
# Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr —
# none of which speak Prometheus natively. The gate would be red for every change until that
# work lands, so it would get disabled, and a disabled gate protects nothing.
# * merely REPORT: that is precisely how these rules stayed inert across phases 4-6.
# * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT
# GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair
# with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted —
# also FAILS, so neither register can rot into fiction.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
# - It does not prove a target is UP, that an endpoint answers, that the chart still honours the
# values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
# recorded in the commit message: with the enablements applied, longhorn_volume_robustness
# returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read
# up==1. A live probe needs a cluster, which L0 by definition does not have.
# - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this
# repo: the port name lives outside the tree. That is why (4) is scoped to same-file
# Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the
# platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their
# liveness is covered by the fact that their metrics have owners under (1).
# - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a
# metric in a scraped family whose exact name the exporter does not emit still passes. This is
# a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}`
# (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the
# series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the
# only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing
# names are the phase_transition_time histograms. Corrected in the same change to
# `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F
# matcher selects nothing and is just as inert). Catching that class needs the exporter's own
# metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that
# diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to
# every LABEL VALUE an alert matches on.
# - The metric-name parser is a PromQL approximation, not a parser: it strips string literals,
# label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as
# metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would
# be missed. No rule in this repo does that, and assertion (0) fails if the parser stops
# finding metrics at all.
# - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the
# L3 alerting test own that).
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
#
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test
# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the
# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2
exit 3
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures.
PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour:
alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the
kind of approximate parser that silently stops matching — the exact failure class this check
exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def read(p: Path) -> str:
try:
return p.read_text()
except (OSError, UnicodeDecodeError):
return ""
# =============================================================================================
# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2).
#
# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose
# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This
# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the
# part that makes the check a REGRESSION guard rather than a one-time audit: every `require`
# regex is re-evaluated against the working tree on every gate run.
#
# `require` — ALL must match the file. The enablement itself.
# `forbid` — NONE may match. For enablements that can be defeated without being deleted.
#
# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_.
# =============================================================================================
KPS = "platform/monitoring/kube-prometheus-stack.yaml"
SCRAPE_OWNERS: list[dict] = [
{
"owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor",
"prefixes": ("node_",),
"file": KPS,
# The subchart block must be present and configured here (it is where the exporter's
# resources live). The chart default is enabled:true, so the regression to guard is a
# DISABLE, not a missing enable.
"require": [r"^\s*prometheus-node-exporter:\s*$"],
"forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
"why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure "
"rules that exist because of prod-1's etcd WAL starvation",
},
{
"owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
# container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as
# kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix
# was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it).
# Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the
# three nodes, job="kubelet", metrics_path="/metrics/cadvisor".
"prefixes": ("kubelet_", "container_"),
"file": KPS,
# Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
# honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
"forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
"why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
"pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the "
"only signal on this cluster that can name WHICH workload an I/O stall is hitting",
},
{
"owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT "
"disabled the way kubeEtcd/kubeProxy are)",
# DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped
# metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's
# server-side disk metrics (etcd_disk_wal_fsync_duration_seconds,
# etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for
# k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for
# {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket
# `etcd_` claim would let a future rule on those names pass this check while being inert,
# which is precisely the hole this test exists to close. Widen only with a real scrape.
"exact": (
"etcd_request_duration_seconds_bucket",
"etcd_request_duration_seconds_count",
),
"file": KPS,
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
"forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
"why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path "
"this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the "
"condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops",
},
{
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
"prefixes": ("kube_velero_schedule_",),
"file": KPS,
"require": [
r"customResourceState:\s*\n\s*enabled:\s*true",
r"metricNamePrefix:\s*kube_velero_schedule",
# The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the
# custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent,
# and the inert-alert failure mode all over again (measured; see the block's header).
r'apiGroups:\s*\["velero\.io"\]',
r'resources:\s*\["schedules"\]',
],
"forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
"why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER "
"backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
"Velero itself publishes nothing until a backup is attempted",
},
{
"owner": "kube-prometheus-stack: bundled kube-state-metrics",
"prefixes": ("kube_",),
"file": KPS,
"require": [r"^\s*kube-state-metrics:\s*$"],
"forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
# kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no
# custom node labels by default, so the metric does not exist unless metricLabelsAllowlist
# names the label. The prefix claim alone would pass while VPNTunnelDown's join matched
# nothing.
"why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule "
"rules",
},
{
"owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)",
"exact": ("kube_node_labels",),
"file": KPS,
"require": [
r"metricLabelsAllowlist:",
# The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
# end in a literal `]`, asserting the list held exactly this one label — so adding a
# second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
# while the property it guards stayed perfectly satisfied. A guard that breaks when you
# correctly extend the thing it guards teaches people to edit the guard, which is how it
# stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
],
"forbid": [],
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
"kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently "
"matches nothing — a rule that is present, evaluating, and structurally inert",
},
{
"owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)",
"prefixes": ("cnpg_",),
"file": "platform/keycloak/postgres-cluster.yaml",
"require": [r"^\s*enablePodMonitor:\s*true\s*$"],
"forbid": [],
"why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog "
"and no-recoverability-point rules that guard R-STATE-03",
},
{
"owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)",
"prefixes": ("gotk_",),
"file": "platform/monitoring/flux-podmonitor.yaml",
"require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"],
"forbid": [],
"why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three "
"automated-update rules — on a platform whose entire delivery model is Flux",
},
{
"owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease",
"prefixes": ("certmanager_",),
"file": "platform/cert-manager/helmrelease.yaml",
"require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"],
# The chart calls `fail` when both monitor kinds are enabled, which would break the whole
# HelmRelease rather than just the scrape. Measured against v1.21.2.
"forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"],
"why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only "
"warning that an ACME DNS-01 renewal of the live wildcard has quietly failed",
},
{
"owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease",
"prefixes": ("longhorn_",),
"file": "platform/longhorn/helmrelease.yaml",
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
"forbid": [],
"why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode "
"is a disk",
# The ServiceMonitor ALONE is not sufficient here — see the companion assertion below.
"companion": {
"file": "platform/longhorn/kustomization.yaml",
"require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"],
"why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies "
"(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on "
"networkPolicies.enabled) whose longhorn-manager policy admits only other "
"Longhorn components in the same namespace. MEASURED on prod: with the "
"ServiceMonitor applied and no allowance, all three targets were DISCOVERED and "
"every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not "
"a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml "
"the four storage alerts are inert even though the monitor exists",
},
},
{
"owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml",
"prefixes": ("kubevirt_",),
"file": "platform/kubevirt/servicemonitor.yaml",
"require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"],
"forbid": [],
"why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use "
"the kubevirt_cdi_* names, not bare cdi_*)",
},
{
"owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease",
"prefixes": ("velero_",),
"file": "platform/velero/helmrelease.yaml",
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
"forbid": [],
"why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
"that failed to notice prod running with zero volume backups",
},
]
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4).
UNIVERSAL = {"up"}
# =============================================================================================
# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and
# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes.
# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert,
# or that gains an owner, FAILS as stale.
# =============================================================================================
KNOWN_UNWIRED_METRICS: dict[str, str] = {
"hubble_drop_total":
"HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression "
"visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and "
"only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on "
"in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} "
"but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on "
"OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes "
"— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; "
"cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — "
"and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same "
"commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled "
"one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's "
"header in platform/monitoring/alerts.yaml.",
}
# =============================================================================================
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only, and NOW EMPTY.
#
# Each entry was a `up{job="X"}` selector whose monitor named an endpoint port the Service it
# selects did not define, so the monitor discovered ZERO targets, no `up{job="X"}` series existed,
# and `up{job="X"} == 0` could never fire. The nastiest variant of the class: `up` exists in
# abundance, so nothing about the metric name looks wrong, and `PrometheusTargetsMissing` cannot see
# it either — a target that was never discovered has no up series to be 0.
#
# IT HELD ALL TWELVE apps/* SLI RULES, AND IT IS EMPTY BECAUSE THEY WERE FIXED, not because the gap
# was waived (2026-09-13, branch fix/app-sli-rules-discover-no-targets). The register's own note
# said closing it needed "a Prometheus exporter per app". That was the wrong conclusion from a true
# premise: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud
# genuinely export no Prometheus metrics, so instead the `up` SLI was rewritten against a signal that
# DOES exist for every workload — kube-state-metrics readiness of the service's own workload object
# (`kube_deployment_status_replicas_available` / `kube_statefulset_status_replicas_ready`), and
# `kubevirt_vmi_info{phase="running"}` for the VM. No exporter, no new scrape target; each rule is
# unit-tested firing AND quiet (apps/wave-*/tests/app-slis.test.yaml) and each expression was
# evaluated live on prod, where eleven of the twelve return no series on a healthy cluster and the
# twelfth (vm-demo) is not deployed at all.
#
# The rendered ServiceMonitors went with them: a monitor for a workload that serves no metrics is
# dead configuration that reads as coverage. So assertion (4) below now has NOTHING to audit under
# apps/ — which is why the monitor-side axis moved into its own check,
# tests/l0-static/servicemonitor-ports-resolve. THAT is where a ServiceMonitor naming a port no
# Service defines fails, with no alert rule required, resolving by selector across the whole tree
# rather than by same-file adjacency, and with its resolver proven on fixtures so it cannot pass for
# lack of subjects. Keeping this register non-empty as "documentation" of a closed gap is exactly the
# fiction the ratchet forbids: the stale-entry assertion below would fail, and rightly.
# =============================================================================================
KNOWN_UNWIRED_JOBS: dict[str, str] = {}
# =============================================================================================
# (0) PARSE EVERY ALERT RULE
# =============================================================================================
# PromQL metric-name approximation. Documented as a gap in the header: string literals,
# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every
# remaining bare identifier that is not a PromQL keyword or function is treated as a metric
# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an
# approximate parser silently stops guarding.
PROMQL_KEYWORDS = set("""
abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week
day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum
histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate
label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn
sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count
count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio
last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time
quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time
double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time
by without on ignoring group_left group_right offset bool and or unless if atan2 start end
inf nan NaN Inf
""".split())
IDENT_RE = re.compile(r"(?<![\w:.])([a-zA-Z_:][a-zA-Z0-9_:]*)")
UP_JOB_RE = re.compile(r"""\bup\s*\{[^}]*\bjob\s*=\s*["']([^"']+)["']""")
def metric_names(expr: str) -> list[str]:
e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)
e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e)
e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e)
e = re.sub(r"\{[^{}]*\}", " ", e) # label matchers
for _ in range(6): # nested aggregations
e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e)
out, seen = [], set()
for m in IDENT_RE.finditer(e):
n = m.group(1)
if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen:
continue
seen.add(n)
out.append(n)
return out
def is_golden_fixture(p: Path) -> bool:
"""conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests.
conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a
rendered `<app>-down` PrometheusRule. They are the assertion data of another test, never
reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the
same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/
fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/.
"""
parts = p.parts
return "conformance" in parts and "golden" in parts
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
rule_files = sorted(
p for p in ROOT.rglob("*.yaml")
if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
)
check(len(rule_files) > 0,
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
"this check would silently audit nothing")
# metric -> [ "<relpath>:<alert>", ... ] ; and the same shape for up{job="X"} selectors
refs: dict[str, list[str]] = {}
job_refs: dict[str, list[str]] = {}
n_rules = 0
for f in rule_files:
rel = str(f.relative_to(ROOT))
try:
docs = list(yaml.safe_load_all(read(f)))
except yaml.YAMLError as exc:
check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited")
continue
for d in docs:
if not isinstance(d, dict) or d.get("kind") != "PrometheusRule":
continue
for grp in (d.get("spec") or {}).get("groups") or []:
for r in grp.get("rules") or []:
if "alert" not in r:
continue # recording rules: not this check's subject
n_rules += 1
site = f"{rel}:{r['alert']}"
expr = str(r.get("expr", ""))
for n in metric_names(expr):
refs.setdefault(n, []).append(site)
for j in UP_JOB_RE.findall(expr):
job_refs.setdefault(j, []).append(site)
check(n_rules > 0,
"no alerting rule parsed out of any PrometheusRule — the parse contract broke and this "
"check would pass vacuously while every alert's metric went unaudited")
check(len(refs) >= 20,
f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The "
f"PromQL approximation has stopped matching — which is exactly how an approximate parser "
f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it "
f"with a real parser")
# =============================================================================================
# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO
# =============================================================================================
def resolve(metric: str) -> dict | None:
"""Longest-prefix / exact-name match against SCRAPE_OWNERS."""
best, best_len = None, -1
for o in SCRAPE_OWNERS:
if metric in o.get("exact", ()): # exact names win outright
return o
for pfx in o.get("prefixes", ()):
if metric.startswith(pfx) and len(pfx) > best_len:
best, best_len = o, len(pfx)
return best
# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing
# at a deleted file is a claim nothing backs.
for o in SCRAPE_OWNERS:
check((ROOT / o["file"]).is_file(),
f"SCRAPE_OWNERS names {o['file']} as the scrape owner for "
f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the "
f"claim is unbacked and the alerts that depend on it are inert")
claimed: dict[str, dict] = {}
unowned: list[str] = []
for metric in sorted(refs):
if metric in UNIVERSAL:
continue
o = resolve(metric)
if o is None:
if metric in KNOWN_UNWIRED_METRICS:
continue # declared gap; ratcheted below
unowned.append(metric)
check(False,
f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is "
f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape "
f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no "
f"vector, stays silent forever, and is indistinguishable from a healthy result. "
f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — "
f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported "
f"instead of silent")
continue
check(metric not in KNOWN_UNWIRED_METRICS,
f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by "
f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is "
f"the one that shows up")
claimed[metric] = o
# The owners that are actually load-bearing right now get their declarations verified.
for o in SCRAPE_OWNERS:
if o not in claimed.values():
continue # nothing queries it today; nothing to guard
body = read(ROOT / o["file"])
for rx in o.get("require", []):
check(re.search(rx, body, re.M) is not None,
f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. "
f"Removing it re-inerts every alert on "
f"{o.get('prefixes') or o.get('exact')} — {o['why']}")
for rx in o.get("forbid", []):
check(re.search(rx, body, re.M) is None,
f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} "
f"even though the enablement is still authored. {o['why']}")
comp = o.get("companion")
if comp:
cbody = read(ROOT / comp["file"])
check((ROOT / comp["file"]).is_file(),
f"{comp['file']} is missing — {comp['why']}")
for rx in comp.get("require", []):
check(re.search(rx, cbody, re.M) is not None,
f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: "
f"{comp['why']}")
# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the
# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off.
NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml"
if any(m.startswith("longhorn_") for m in refs):
np = read(NP)
check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its "
f"four alerts are inert")
check(re.search(r"kind:\s*NetworkPolicy", np) is not None,
"platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy")
check(re.search(r"app:\s*longhorn-manager", np) is not None,
"the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not "
"open the pods that serve /metrics")
check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None,
"the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus "
"still cannot reach longhorn-manager and the storage alerts stay inert")
check(re.search(r"port:\s*9500", np) is not None,
"the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` "
"port, which serves /metrics)")
# LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports.
check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None
and "Egress" not in np,
"the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the "
"scrape path and nothing else (R-SEC-02)")
# Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that
# they stay closed, and a substring search over prose would fail on the documentation of the
# very property it is asserting.
np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines())
for closed in ("9502", "9503"):
check(closed not in np_spec,
f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / "
f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring "
f"is reach this observability path does not need (R-SEC-02)")
# stale KNOWN_UNWIRED_METRICS entries
for m in sorted(KNOWN_UNWIRED_METRICS):
check(m in refs,
f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any "
f"more — a stale exemption hides the next real gap; delete the entry")
# =============================================================================================
# (4) up{job="X"} — the axis assertion (1) is structurally blind to.
#
# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/*
# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose
# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap).
# =============================================================================================
def yaml_docs(path: Path) -> list[dict]:
try:
return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)]
except yaml.YAMLError:
return []
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
for p in sorted(ROOT.rglob("*.yaml")):
if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
continue
txt = read(p)
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
continue
docs = yaml_docs(p)
monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"]
services = [d for d in docs if d.get("kind") == "Service"]
if not monitors or not services:
continue
svc_ports: set[str] = set()
for s in services:
for port in (s.get("spec") or {}).get("ports") or []:
if isinstance(port, dict) and port.get("name"):
svc_ports.add(str(port["name"]))
for m in monitors:
ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or []
if isinstance(e, dict) and e.get("port")}
name = (m.get("metadata") or {}).get("name")
if name:
same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports)
for job in sorted(job_refs):
sites = ", ".join(sorted(set(job_refs[job])))
mon = same_file_monitors.get(job)
if mon is None:
# No same-file Service+ServiceMonitor pair is named for this job. It may well be produced
# by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all
# are), which this check cannot see — so this is NOT a failure, only unauditable.
continue
rel, ep_ports, svc_ports = mon
missing = sorted(ep_ports - svc_ports)
if missing:
check(job in KNOWN_UNWIRED_JOBS,
f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor "
f"scrapes port(s) {missing}, and the Service(s) in that same file define only "
f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO "
f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is "
f"structurally inert — note PrometheusTargetsMissing cannot see this either, because "
f"a target that was never discovered has no up series to be 0. Give the Service a "
f"named port the workload actually serves metrics on (and an exporter if it has "
f"none), or declare it in KNOWN_UNWIRED_JOBS")
else:
check(job not in KNOWN_UNWIRED_JOBS,
f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) "
f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: "
f"delete the entry")
# stale KNOWN_UNWIRED_JOBS entries
for job in sorted(KNOWN_UNWIRED_JOBS):
check(job in job_refs,
f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any "
f"more — a stale exemption hides the next real gap; delete the entry")
# =============================================================================================
# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves —
# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used
# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and
# HONOURED by ci/l0.sh's --include-scope union.
# =============================================================================================
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo"
SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets"
/ "chainsaw-test.yaml")
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
"alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare "
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
"change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and "
"platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a "
"chart's serviceMonitor.enabled back to false would be filtered out of exactly the change "
"that re-inerts the alerts")
L0 = read(ROOT / "ci" / "l0.sh")
GATE = read(ROOT / "ci" / "gate.sh")
check("--include-scope" in L0,
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
"purely requirement-scoped run again and the scope annotation above is inert decoration")
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
and re.search(r"fail\).*BLOCK=1", GATE) is not None,
"ci/gate.sh does not set the merge block when the l0 stage fails")
# ======================================================================== report
owners_used = {id(o): o for o in claimed.values()}
print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} "
f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)")
print(f" claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}"
f" owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}")
for o in SCRAPE_OWNERS:
ms = sorted(m for m, oo in claimed.items() if oo is o)
mark = " " if ms else "· "
print(f" {mark}{o['file']:<46} {len(ms):>2} metric(s) {o['owner'][:64]}")
print(f"\n DECLARED GAPS (shrink-only registers):")
print(f" KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — "
f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}")
print(f" KNOWN_UNWIRED_JOBS: {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — "
f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}")
if unowned:
print(f"\n UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr)
print()
if not FAILURES:
print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)")
raise SystemExit(0)
print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
@@ -1,95 +0,0 @@
# =============================================================================
# L0 STATIC — the alert rules are UNIT-TESTED, by a pinned engine, and the gaps are declared
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# R-OBS-04 ("the platform alerts on its own health") is implemented by PrometheusRule CRs, and
# this repo has shipped both ways they fail SILENTLY:
# * PERMANENTLY FIRING — the CNPG primaries alert (fixed 2026-09-12), VPNTunnelDown before it;
# * STRUCTURALLY INERT — VeleroNoRecentBackup, whose `time() - <absent series>` yields no
# series, so a schedule that had NEVER backed up could not raise it. Prod ran with zero
# volume backups and nothing alerted, masked by a resources-only schedule completing
# cleanly every night (2026-09-12).
# An inert rule and a correct rule are indistinguishable while the platform is healthy. Only
# `promtool test rules` separates them, by constructing the bad state on purpose, offline.
#
# The unit tests were then written — and RUN BY NOTHING. promtool was not in the pinned
# toolchain, no gate stage evaluated the files, and the two harnesses that grew up independently
# each fetched an engine off the network mid-run if promtool was missing (a GitHub release
# download in one, a prometheus container image in the other). Coverage that looks present and
# is enforced nowhere is the failure mode this test ends.
#
# GIVEN the repository as authored,
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
# COMMITTED PrometheusRule CRs into the rules file promtool wants — never a copy) is run
# over every test file in the repo with the PINNED promtool,
# THEN promtool is pinned in BOTH versions.env files in lockstep at the Prometheus version prod
# actually runs, baked into the image and installed on the CI runner; the promtool in use
# reports exactly that version; no harness acquires an engine at run time; there is
# exactly ONE harness; every rule-unit test file PASSES; the harness is proven able to
# FAIL by a mutated synthetic test on every run; and every alert rule with NO unit test is
# ENUMERATED in a shrink-only register, so the gap is reported rather than hidden.
#
# MISSING TESTS: RATCHET, NOT SKIP, NOT BLANKET-FAIL. 48 of 55 alert rules have no unit test
# today. Failing on all of them would make the gate red until ~48 tests exist (so it would be
# disabled, and a disabled gate protects nothing); merely reporting is precisely how the CNPG
# tests came to be enforced by nothing. So the gap is DECLARED in check.sh, printed with counts
# every run, and CANNOT GROW: a new PrometheusRule, or a new alert in an already-tested file,
# FAILS without a test, and a stale register entry FAILS too. A missing test is never an exit-3
# skip — "SKIPPED is not green" is preserved for the one thing that is genuinely an environment
# gap: promtool being absent.
#
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (standard library + the harness
# it delegates to); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# NOT proven (named in check.sh's header): that an alert reaches a human (Alertmanager routing
# and SMTP are L3 / R-OBS-02), that a threshold is the right one, that a rule file is actually
# reconciled into a cluster, or that an author's assertions are meaningful — coverage of one
# alert is detected by its name appearing in a covering test file, and only the mutation
# self-test proves the harness CAN fail. R-OBS-04 is tagged for the static slice only: the rules
# the repo ships behave, under the engine prod runs, as their own unit tests say.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-rules-unit-tested
labels:
conformance.home.k3s/R-OBS-04: "true"
conformance.home.k3s/R-TEST-02: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04, R-TEST-02"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree
# (apps/wave-0..2, platform/cnpg, platform/kubevirt, platform/monitoring), and the register
# it maintains is a property of the WHOLE repo, not of any one path.
#
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map keys
# apps/* to NOTHING and platform/monitoring/* to R-OBS-01/02 — not R-OBS-04. So adding a new
# untested PrometheusRule under apps/, or editing one under platform/kubevirt/, attributes
# no requirement this test carries, and a requirement-scoped selection would filter the
# check out of exactly the change that adds the untested alert. The same blind spot that let
# `prod-baremetal-not-dev-surface` be filtered out of the run that gated its own breakage
# (DECISIONS-LOG 2026-09-12).
#
# Declaring the scope also makes an honest SKIP loud: lib/run_l1_offline.py turns a rc-3
# skip of a scope-declaring check into the runner's rc 3, and ci/l0.sh into the gate's rc 3.
# So a gate run on a toolchain WITHOUT promtool reports UNWIRED instead of green — which is
# the whole reason this check exists rather than another hand-written ci/l0.sh stage.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every committed alert rule that has unit tests passes them under the pinned promtool that
matches prod's Prometheus, exactly one harness runs them, nothing fetches a rule engine at
run time, the harness is proven able to fail, and every alert rule without a unit test is
enumerated in a register that can only shrink.
spec:
steps:
- name: run-every-alert-rule-unit-test-and-report-the-gaps
description: >-
Assert the promtool pin is in lockstep and baked, run the single rule-unit harness over
every test file in the repo, prove it can fail via a mutated synthetic test, and
enumerate every PrometheusRule with no unit tests against the shrink-only register.
try:
- script:
content: bash ./check.sh
@@ -1,580 +0,0 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — alert-rules-unit-tested
# Serves: R-OBS-04 ("Platform alerts on its own health (nodes, storage, GitOps, certs,
# VPN)." — spec docs/10-observability-and-alerting.md)
# R-TEST-02 ("Tests are reproducible: deterministic, hermetic, single-command."
# — spec docs/14-testing-and-conformance.md)
#
# WHY THIS EXISTS — AN UNTESTED ALERT IS NOT A MONITORING CONTROL
# --------------------------------------------------------------
# This repo has shipped BOTH halves of the alerting failure space to prod:
# * a PERMANENTLY-FIRING `critical` (the CNPG alert fixed 2026-09-12; VPNTunnelDown before
# it), and
# * a STRUCTURALLY INERT rule — VeleroNoRecentBackup, whose `time() - <absent series>`
# yields no series, so a schedule that had NEVER backed up could not raise it. Prod ran
# with no volume backups at all and nothing alerted (2026-09-12).
# Neither is visible from the expression, from `kubectl apply`, or from a healthy cluster: an
# inert rule and a correct rule are indistinguishable while everything is fine. `promtool test
# rules` is the only tool that separates them, because the BAD state can be constructed on
# purpose. Unit tests for those rules were then WRITTEN — and evaluated by nothing: promtool
# was not in the pinned toolchain, no gate stage ran the files, and each of the two harnesses
# that existed acquired an engine off the network mid-run if one was missing. Coverage that
# looks present and is never enforced is the failure mode this check ends.
#
# GIVEN the repository as authored,
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
# COMMITTED PrometheusRule CRs — never a copy) is asked which manifests are under test
# and then run over every test file in the repo,
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
# CI runner installs it for the l0 stage;
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
# evidence only if the engine matches the one prod evaluates rules with;
# (3) NO harness acquires a rule engine at RUN TIME — neither `curl | tar -xz` of a
# GitHub release nor a `podman/docker run` of a prometheus image. Both existed on
# 2026-09-12 and both are removed; this is the regression guard;
# (4) there is exactly ONE harness. Two were written independently within an hour of
# each other; the one nothing runs is the one that rots;
# (5) every rule-unit test file in the repo PASSES, run through that harness (which
# `promtool check rules` first, so a non-parsing rules file cannot make unit tests
# pass vacuously by finding no rules at all);
# (6) the harness is NOT VACUOUS: a synthetic PrometheusRule + companion test is run
# through the SAME harness twice — intact (must PASS) and with one expectation
# mutated (must FAIL) — so a harness that has silently stopped evaluating anything
# is caught on every run, not only when someone thinks to try it;
# AND (7) the alert rules with NO unit tests are ENUMERATED and RATCHETED: each is declared
# below, printed with counts on every run, and the register can only SHRINK.
#
# WHY A MISSING TEST IS A RATCHET AND NOT AN OUTRIGHT FAIL (the decision, and its reasons)
# ---------------------------------------------------------------------------------------
# 48 of the repo's 55 alert rules have no unit test today. Three options were on the table:
# * FAIL on any missing test — correct in principle, but it makes the merge gate red for
# every change until ~48 unit tests exist, so in practice the gate gets disabled, and a
# disabled gate protects nothing.
# * merely REPORT — the gap prints and nothing holds it. That is exactly how the CNPG unit
# tests came to exist and be enforced by nothing; "it is in the output somewhere" is not
# enforcement.
# * RATCHET (chosen) — the gap is DECLARED IN THE REPO as a reviewable register, printed with
# counts on every gate run, and it CANNOT GROW: a new PrometheusRule, or a new alert in a
# file that already has tests, FAILS unless it ships a unit test. Removing an entry requires
# writing the test, and a STALE entry (file gone, or now covered) also FAILS, so the
# register cannot rot into fiction.
# "SKIPPED is not green" is honoured: a missing test is NEVER an exit-3 skip here. This check
# exits 0 only with the gap fully enumerated, and exits 1 the moment the debt grows, the
# register drifts, or a test that exists stops passing.
# The one exit-3 path is promtool itself being ABSENT — an environment gap, not a code property
# (the same convention as ci/l0.sh's kyverno handling), and this test DECLARES
# `conformance.home.k3s/scope: whole-repo`, which makes lib/run_l1_offline.py turn that skip
# into the runner's rc 3 and ci/l0.sh into the gate's rc 3: UNWIRED, never green. promtool
# PRESENT AT THE WRONG VERSION is a FAILURE, not a skip: that case still prints SUCCESS while
# evaluating the rules with an engine prod does not run.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps):
# - It does not prove an alert reaches a human. promtool evaluates rules against synthetic
# series in-process; Alertmanager routing, SMTP delivery and inhibition are L3
# (tests/l3-fault-injection/alerting) and R-OBS-02, and are not claimed here.
# - It does not prove the THRESHOLDS are right, only that a rule behaves as its own unit
# test says.
# - It does not prove a rule file is RECONCILED into a cluster. An alert file no
# kustomization includes is a separate "looks applied and isn't" failure, not audited here.
# - Coverage of an individual alert is detected by its NAME appearing in a test file that
# covers its manifest. A test file that merely mentions a name without asserting anything
# counts as covered; only the mutation self-test (6) proves the harness CAN fail, not that
# every author wrote a meaningful assertion. Review still owns test quality.
# - It does not read the live cluster, so "the pin matches prod's Prometheus" is asserted
# against the repo's pin, which was established by a live read recorded in
# infra/devcontainer/versions.env (2026-09-12: v3.14.0-distroless).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
HARNESS="${ALERT_RULE_HARNESS:-${ROOT}/platform/monitoring/tests/run.sh}"
# Honest SKIP (never green) if the thing under test is not present at all. Because this test
# declares scope: whole-repo, the runner turns this into rc 3 for the whole L0 stage — a
# PrometheusRule-less repo is a vanished tree, not a pass.
if ! grep -rqls --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
"${ROOT}/platform" "${ROOT}/apps" 2>/dev/null; then
echo "check SKIPPED: no PrometheusRule under platform/ or apps/ — nothing to test" >&2
exit 3
fi
# promtool resolution. PROMTOOL= overrides (the same override the harness honours); otherwise
# PATH. Absent -> every OFFLINE assertion still runs and the check exits 3, so a toolchain gap
# is reported as a gap and never as a pass.
PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
if [ -n "${PROMTOOL_BIN}" ] && [ -x "${PROMTOOL_BIN}" ]; then
PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
else
PROMTOOL_BIN=""
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" "${HARNESS}" "${PROMTOOL_BIN}" <<'PY'
"""Assert the alert-rule unit tests are pinned, enforced, non-vacuous, and that the alert
rules with no tests are enumerated and ratcheted. Standard library + the harness itself."""
from __future__ import annotations
import os
import re
import shutil
import subprocess
import sys
import tempfile
from collections import defaultdict
from pathlib import Path
ROOT = Path(sys.argv[1])
HARNESS = Path(sys.argv[2])
PROMTOOL = sys.argv[3]
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
# =============================================================================
# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
#
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
#
# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
# generated from one ServiceDefinition template, and the register's description of them as one
# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
# now the only entry left and is where the next tests belong.
# =============================================================================
UNTESTED_FILES = {
"platform/kubevirt/alerts.yaml",
}
# UNTESTED_ALERTS — individual alerts inside a file that IS covered by a test file, but whose
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
# already-"tested" file untested: anything not listed here must appear in a covering test.
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
# redundancy meant asserting on the rules that survive). Only
# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
# It SHRANK BY TWO MORE on 2026-09-14: platform/monitoring/tests/node-io-cause.test.yaml pins the
# eight new platform-health.node-io-cause rules, and its healthy fixture asserts NodeIOPressureHigh
# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
# detectors are no longer unasserted.
# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
# had to become an assertion rather than a sentence in a comment.
UNTESTED_ALERTS = {
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
("platform/cnpg/alerts.yaml", "CNPGNoRecentBaseBackup"),
("platform/cnpg/alerts.yaml", "CNPGWALVolumeNotRecycling"),
("platform/cnpg/alerts.yaml", "CNPGMetricsCollectorErroring"),
("platform/monitoring/alerts.yaml", "NodeExporterDown"),
("platform/monitoring/alerts.yaml", "NodeMemoryPressureHigh"),
("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
("platform/monitoring/alerts.yaml", "KeycloakDown"),
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
}
# --------------------------------------------------------------- (1) the pins, in lockstep
def env_pin(rel: str, key: str) -> str | None:
f = ROOT / rel
if not f.is_file():
return None
m = re.search(rf'^{re.escape(key)}=["\']?([^"\'\s#]+)', f.read_text(), re.M)
return m.group(1) if m else None
DEV_PIN = env_pin("infra/devcontainer/versions.env", "PROMTOOL_VERSION")
CI_PIN = env_pin("ci/versions.env", "PROMTOOL_VERSION")
check(DEV_PIN is not None,
"infra/devcontainer/versions.env declares no PROMTOOL_VERSION — promtool is not part of the "
"baked toolchain, so the alert unit tests are evaluated by whatever the host happens to "
"have, or by nothing at all (which is where this repo was on 2026-09-12)")
check(CI_PIN is not None,
"ci/versions.env declares no PROMTOOL_VERSION — the gate's own version file does not pin the "
"rule engine, so CI and the devcontainer can silently diverge")
check(DEV_PIN is None or DEV_PIN == CI_PIN,
f"PROMTOOL_VERSION is NOT in lockstep: infra/devcontainer/versions.env={DEV_PIN!r} vs "
f"ci/versions.env={CI_PIN!r}. The repo's rule is that the conformance-tool versions in the "
f"two files stay identical, so a laptop and CI run the same engine")
CF = ROOT / "infra/devcontainer/Containerfile"
cf_text = CF.read_text() if CF.is_file() else ""
check(re.search(r"^ARG PROMTOOL_VERSION\s*$", cf_text, re.M),
"infra/devcontainer/Containerfile has no `ARG PROMTOOL_VERSION` — the pin is declared but "
"the image build does not consume it")
check("prometheus/prometheus/releases/download" in cf_text and "promtool" in cf_text,
"infra/devcontainer/Containerfile does not install promtool from the pinned prometheus "
"release tarball (promtool ships inside it)")
check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
"that provenance line is how an operator tells a stale image from a current one")
DEVSH = ROOT / "infra/devcontainer/dev.sh"
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
"empty and the build would fetch a bogus URL")
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
if INSTALL.is_file():
itext = INSTALL.read_text()
check("PROMTOOL_VERSION" in itext
and re.search(r"^\s*l0\)[^\n]*install_promtool", itext, re.M),
".gitea/workflows/scripts/install-tools.sh does not install promtool for the l0 stage — "
"on the CI runner this check would exit 3 (UNWIRED) forever and the alert unit tests "
"would be enforced nowhere, which is the exact gap it exists to close")
# --------------------------------------------- (3)(4) one harness, and it fetches no engine
check(HARNESS.is_file() and os.access(HARNESS, os.X_OK),
f"the rule-unit harness is missing or not executable at {HARNESS.relative_to(ROOT)} — "
f"nothing would run the alert unit tests")
# Any script that invokes `promtool test rules` is a rule-unit harness. There must be exactly
# one: two harnesses were written independently on 2026-09-12 (a cnpg-only script and this one),
# and the one nothing runs is the one that rots out of sync with the rules.
harnesses: list[str] = []
fetchers: list[str] = []
FETCH_RE = re.compile(
r"(curl|wget)[^\n]*prometheus/prometheus/releases" # GitHub release download
r"|(podman|docker)\s+run[^\n]*prometheus", # container-image pull
re.I)
# Two exclusions, both the repo's existing "reference, not declaration" rule (see
# ci/touched-requirements.sh changed_diff() rules 2-3):
# * THIS CHECK'S OWN DIRECTORY. It names every forbidden pattern in order to forbid it, so
# scanning itself would make both guards permanently red — the self-referential trap.
# * THE PINNED TOOLCHAIN INSTALLERS. Downloading the pinned promtool is their JOB, at a
# reviewed seam (an image build, or the runner's tool-install step), from a version in
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
# version, the network and the machine are decided silently at test time, and a missing
# toolchain becomes a green run instead of an exit 3.
# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
for p in sorted(ROOT.rglob("*.sh")):
rel = str(p.relative_to(ROOT))
if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
continue
try:
text = p.read_text()
except (OSError, UnicodeDecodeError):
continue
body = "\n".join(l for l in text.splitlines() if not l.lstrip().startswith("#"))
if "promtool" in body and re.search(r"test\s+rules", body):
harnesses.append(rel)
if FETCH_RE.search(body) and not rel.startswith(INSTALLERS):
fetchers.append(rel)
check(harnesses == [str(HARNESS.relative_to(ROOT))],
f"expected exactly ONE rule-unit harness ({HARNESS.relative_to(ROOT)}) but found "
f"{harnesses}. A second harness means two behaviours and two things to keep in step; the "
f"one the gate does not run is the one that rots. Fold it in instead")
check(not fetchers,
f"{fetchers} acquire a rule ENGINE at run time (a prometheus release download or a "
f"prometheus container image). promtool must come from the pinned toolchain "
f"(PROMTOOL_VERSION): a run-time fetch decides the version, the network and the machine "
f"silently at test time, makes the tests non-hermetic (R-TEST-02), and quietly turns 'the "
f"toolchain has no promtool' into a green run — which is how it stayed missing")
# ------------------------------------------------------------------ discover the alert files
# Alert files: tracked YAML carrying a PrometheusRule. conformance/ is excluded deliberately —
# the PrometheusRules under tests/l1-rendering/*/golden/ are EXPECTED-RENDER fixtures for the
# render checks, not rules any cluster evaluates. Everything a cluster actually gets lives under
# apps/, platform/, platform-config/ or clusters/.
def tracked_yaml() -> list[Path]:
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
capture_output=True, text=True)
if out.returncode == 0 and out.stdout:
return [ROOT / p for p in out.stdout.split("\0") if p]
# No usable index (the gate container mounts a worktree whose .git points outside the
# mount). Fall back to a filesystem walk rather than auditing nothing.
return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
ALERT_RE = re.compile(r"^\s*-\s*alert:\s*(\S+)", re.M)
alert_files: dict[str, list[str]] = {}
for p in tracked_yaml():
rel = str(p.relative_to(ROOT))
if rel.startswith("conformance/") or not p.is_file():
continue
try:
text = p.read_text()
except (OSError, UnicodeDecodeError):
continue
if KIND_RE.search(text):
alert_files[rel] = ALERT_RE.findall(text)
check(len(alert_files) > 0,
"no PrometheusRule found outside conformance/ — discovery or the repo layout changed, and "
"this check would silently audit nothing")
# ------------------- ask the HARNESS which manifests are under test (never re-derive it here)
# Two implementations of "which rules does this test cover" would eventually disagree, and the
# disagreement shows up as a file the gate believes is tested and the harness never touches.
listing = subprocess.run(["bash", str(HARNESS), "--list"], capture_output=True, text=True,
cwd=str(ROOT))
check(listing.returncode == 0,
f"`{HARNESS.relative_to(ROOT)} --list` failed (rc={listing.returncode}): "
f"{(listing.stderr or listing.stdout).strip()[-600:]}")
covering: dict[str, list[str]] = defaultdict(list) # manifest -> [test files]
for line in listing.stdout.splitlines():
if "\t" in line:
tfile, manifest = line.split("\t", 1)
covering[manifest.strip()].append(tfile.strip())
for manifest in sorted(covering):
check(manifest in alert_files,
f"the harness reports {manifest} as under test, but it is not one of the "
f"PrometheusRule files found in the repo — the mapping and the tree disagree")
# --------------------------------------------------------- (2) the engine is the pinned one
version_line = ""
if PROMTOOL:
vp = subprocess.run([PROMTOOL, "--version"], capture_output=True, text=True)
out = (vp.stdout + vp.stderr).strip()
version_line = out.splitlines()[0] if out else ""
m = re.search(r"version\s+(\d+\.\d+\.\d+)", version_line)
found = m.group(1) if m else None
check(found is not None,
f"could not parse a version from `promtool --version`: {version_line!r}")
check(found is None or DEV_PIN is None or found == DEV_PIN,
f"promtool reports {found}, but the toolchain pins PROMTOOL_VERSION={DEV_PIN} (the "
f"Prometheus version prod runs). promtool IS the rule engine: a unit test green under "
f"one version and red under another is worse than no test. Run the gate in the pinned "
f"devcontainer (rebuild it: ./infra/devcontainer/dev.sh build), or set PROMTOOL= to a "
f"v{DEV_PIN} binary. A FAILURE, not a skip, because the wrong engine still prints "
f"SUCCESS")
# ------------------------------------------------- (5) every rule-unit test file must pass
if PROMTOOL:
env = dict(os.environ, PROMTOOL=PROMTOOL)
run = subprocess.run(["bash", str(HARNESS)], capture_output=True, text=True,
cwd=str(ROOT), env=env)
tailed = "\n ".join((run.stdout + run.stderr).strip().splitlines()[-40:])
check(run.returncode == 0,
f"`{HARNESS.relative_to(ROOT)}` FAILED (rc={run.returncode}) — the committed alert "
f"rules do not behave as their own unit tests say:\n {tailed}")
# ------------------------------------------- (6) the harness is proven non-vacuous
# A synthetic PrometheusRule + companion test through the SAME harness, outside the repo so
# discovery never picks it up. Intact must PASS; a mutated expectation must FAIL. Without
# this, a harness that stopped evaluating anything (an empty extraction, a rule file written
# under the wrong name) would report green forever — which is the disease being treated.
work = tempfile.mkdtemp(prefix="alert-rules-selftest.")
try:
(Path(work) / "synth.yaml").write_text("""apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: selftest
spec:
groups:
- name: selftest
rules:
- alert: SelfTestCanary
expr: selftest_metric > 0
for: 5m
annotations:
runbook_url: https://${BASE_DOMAIN}/runbooks/selftest
""")
test_yaml = """rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
- interval: 1m
name: canary fires only after `for` elapses
input_series:
- series: 'selftest_metric'
values: '1x10'
promql_expr_test:
- expr: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
eval_time: EVAL_TIME
exp_samples:
- labels: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
value: 1
"""
synth_test = Path(work) / "synth.promtool-test.yaml"
synth_test.write_text(test_yaml.replace("EVAL_TIME", "6m"))
good = subprocess.run(["bash", str(HARNESS), str(synth_test)],
capture_output=True, text=True, cwd=str(ROOT), env=env)
check(good.returncode == 0,
"self-test: the INTACT synthetic unit test did not pass, so the harness itself is "
"broken and every green result above is meaningless:\n "
+ "\n ".join((good.stdout + good.stderr).strip().splitlines()[-20:]))
# Mutation: assert firing at 2m, which `for: 5m` forbids.
synth_test.write_text(test_yaml.replace("EVAL_TIME", "2m"))
bad = subprocess.run(["bash", str(HARNESS), str(synth_test)],
capture_output=True, text=True, cwd=str(ROOT), env=env)
check(bad.returncode != 0,
"self-test: a MUTATED expectation (firing before `for:` elapsed) still PASSED — "
"the harness is not actually evaluating anything, so every green result above is "
"vacuous")
finally:
shutil.rmtree(work, ignore_errors=True)
# --------------------------------------------------------- (7) the gap inventory + ratchet
INVENTORY: list[str] = []
GAP_FILES: list[str] = []
alerts_total = covered_total = 0
for rel in sorted(alert_files):
names = alert_files[rel]
alerts_total += len(names)
check(len(names) > 0,
f"{rel} declares a PrometheusRule but no `alert:` rule was parsed out of it — the "
f"parser or the file shape changed, and any unit test over it would pass vacuously")
tests = covering.get(rel, [])
if not tests:
GAP_FILES.append(rel)
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested=0/{len(names):<3} NO TEST FILE")
check(rel in UNTESTED_FILES,
f"{rel} carries {len(names)} alert rule(s) and NO test file covers it. A new "
f"PrometheusRule must ship promtool unit tests: this repo has shipped both a "
f"permanently-firing critical and a structurally inert alert, and neither was "
f"visible without them. Add `{rel[:-5]}.promtool-test.yaml` (companion convention) "
f"or a <component>/tests/*.test.yaml suite. If this is deliberate debt it must be "
f"DECLARED in UNTESTED_FILES in this check — the register is what keeps the gap "
f"visible instead of silent")
continue
check(rel not in UNTESTED_FILES,
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
f"shrink-only: delete the line so the next missing test is caught")
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
# "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
# This used to be a bare word match over the whole test file, so an alert named anywhere —
# including in a PROSE COMMENT — counted as tested. Hit for real while adding
# platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
# stays quiet during an egress-steering failure, and that sentence alone was enough to make this
# check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
# the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
# outcome this check exists to prevent, produced BY the check.
# Now an alert counts only when a test file names it in an `alertname:` position, with comment
# lines stripped first so a commented-out assertion does not count either.
# MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
# asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
# registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
# mattered because the registers are shrink-only: had any alert been prose-only, tightening would
# have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
# it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
text_no_comments = "\n".join(
l for l in text.split("\n") if not l.lstrip().startswith("#")
)
# BOTH promtool idioms count, because both are real assertions:
# alert_rule_test: - alertname: CNPGPrimariesCoLocated
# promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
# Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
# platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
# form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
# already better covered than most.
covered = [n for n in names
if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
covered_total += len(covered)
for n in names:
if n in covered:
check((rel, n) not in UNTESTED_ALERTS,
f"{rel}: alert {n} is listed in UNTESTED_ALERTS but IS asserted on now — "
f"remove the stale entry (the register is shrink-only)")
else:
check((rel, n) in UNTESTED_ALERTS,
f"{rel}: alert {n} has NO unit test in {tests}. This file is already covered, "
f"so a new alert in it must be tested too — otherwise a 'tested' alert file "
f"quietly grows untested rules")
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested={len(covered)}/{len(names):<3} "
f"{','.join(tests)}")
# stale register entries
for rel in sorted(UNTESTED_FILES - set(alert_files)):
check(False, f"UNTESTED_FILES lists {rel}, which carries no PrometheusRule in this repo any "
f"more — a stale exemption hides the next real gap; delete the line")
for rel in sorted({f for f, _ in UNTESTED_ALERTS} - set(alert_files)):
check(False, f"UNTESTED_ALERTS lists alerts in {rel}, which carries no PrometheusRule in this "
f"repo any more — delete those lines")
# --------------------------------------------------------------------------------- report
print("Alert-rule unit-test inventory (PrometheusRule -> covering test file(s)):")
for line in INVENTORY:
print(line)
print(f"\nharness: {HARNESS.relative_to(ROOT)} promtool: {version_line or 'NOT FOUND'}")
print(f"pinned: PROMTOOL_VERSION={DEV_PIN} (devcontainer) / {CI_PIN} (ci) — prod runs Prometheus "
f"v{DEV_PIN}")
print(f"alert files: {len(alert_files)} covered by a test file: "
f"{len(alert_files) - len(GAP_FILES)} with NO test file: {len(GAP_FILES)}")
print(f"alert rules: {alerts_total} unit-tested: {covered_total} "
f"NOT unit-tested: {alerts_total - covered_total}")
if GAP_FILES:
print("\nDECLARED GAPS — alert files nothing tests at all (shrink-only register; each is a")
print("rule nothing has ever evaluated, NOT a rule known to work):")
for g in GAP_FILES:
print(f" - {g} ({len(alert_files[g])} alert(s))")
if UNTESTED_ALERTS:
print("\nDECLARED GAPS — individual alerts inside a covered file that nothing asserts on:")
for f, n in sorted(UNTESTED_ALERTS):
print(f" - {f} {n}")
print()
if not PROMTOOL and not FAILURES:
print("check SKIPPED (exit 3, NOT a pass): promtool is not on PATH, so NO alert rule was "
"parsed and NO unit test was evaluated.", file=sys.stderr)
print(f" promtool is pinned at v{DEV_PIN} in infra/devcontainer/versions.env + "
f"ci/versions.env.", file=sys.stderr)
print(" Run the gate in the pinned toolbox (./infra/devcontainer/dev.sh build, then "
"make -C ci gate), or set PROMTOOL=/path/to/promtool.", file=sys.stderr)
print(f" The {CHECKS} offline assertions above (pins, lockstep, one harness, no run-time "
f"engine fetch, the gap register) DID run and passed.", file=sys.stderr)
print(" This test declares scope: whole-repo, so the runner reports rc 3 — UNWIRED, never "
"green.", file=sys.stderr)
raise SystemExit(3)
if not FAILURES:
print(f"check OK: alert-rules-unit-tested — {CHECKS} assertions "
f"({covered_total}/{alerts_total} alert rules unit-tested, "
f"{len(GAP_FILES)} files with no test file, all declared) (R-OBS-04, R-TEST-02)")
raise SystemExit(0)
print(f"CHECK FAIL (alert-rules-unit-tested): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
@@ -1,89 +0,0 @@
# =============================================================================
# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Every platform-health alert rule in this repo carries
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
# stopped matching.
#
# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
# and "the URL resolves" are different claims; only the second one helps an operator.
#
# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
# WHEN each is parsed out of git and resolved against runbooks/,
# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
# (2) every referenced <name> has a runbooks/<name>.md in git;
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
# make the link look fixed while still helping nobody;
# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
# configMapGenerator under the extensionless key the URL asks for;
# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
# the Gateway, and an nginx location for that same prefix with a text/plain default_type);
# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
# substitution would run envsubst over the runbook prose and silently empty every `${...}`
# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
# scans files reachable through `resources:`, not configMapGenerator inputs;
# AND (7) the check is LOAD-BEARING: this Test DECLARES
# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
# script.
#
# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
# runbook links that resolve to real, served procedure rather than to a 404.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: alert-runbooks-resolvable
labels:
conformance.home.k3s/R-OBS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-OBS-04"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
# `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
# annotation into the selection regardless of which requirements the change touched. That union is
# load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
# NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
# likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
# configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
# of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
# per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
floor, and is actually projected into the server that answers that path — so an alert can never
hand an operator a dead link.
spec:
steps:
- name: audit-every-alert-runbook-link
description: >-
Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
served at that exact path, and that the serving tier is wired and unsubstituted.
try:
- script:
content: bash ./check.sh
Loaded 100 of 289 files, more files were not shown because too many files have changed in this diff. Show more